MKE.dev

Milwaukee's zoning code is 700 pages buried across city websites. MKE.dev turns it into one AI conversation -- ask, upload plans, visualize buildings -- powered by Gemini 3.

Inspiration

As a City Plan Commissioner for Milwaukee, I see firsthand how residents, developers, and architects struggle to access the zoning and development information they need. The current experience is painful: jump between multiple city department websites, download PDFs of a 700-page zoning code, search a slow GIS map that was built for planners not the public, manually cross-reference parcel data with code requirements, and hunt through scattered pages to find the right permit form. A homeowner wondering if they can add an ADU faces the same information maze that professional developers hire consultants to navigate.

We wanted to create a completely new way to interact with city services -- one interface where citizens, developers, architects, and planners can all get what they need without the old web navigation runaround. No more downloading PDFs. No more bouncing between departments. No more guessing which code section applies. Gemini 3 gave us the tools to make that possible: ask a question and get a grounded answer, upload a site plan and get a compliance report, look at a vacant lot and see what could be built there -- all in one conversation.

What it does

MKE.dev is an AI-powered civic intelligence platform for Milwaukee that combines:

• Zoning Interpreter Agent -- A Gemini 3 Flash agent with 22 function-calling tools that answers complex zoning questions in real time. Ask "I want to open a restaurant at 2000 N MLK Drive -- what's the zoning, how many parking spaces do I need, and what permits should I file?" and the agent geocodes the address, queries the city GIS, calculates parking, searches the zoning code, and recommends permit forms -- all in one turn.

• Document Analyzer -- Upload a site plan, floor plan, or variance application and Gemini 3 Flash classifies the document type using MINIMAL thinking, auto-enriches it with real zoning and parcel data from Milwaukee's ESRI GIS, then performs a deep compliance analysis using HIGH thinking. The result is a structured report that covers dimensional compliance, setback checks, use compatibility, variance requirements, and risk assessment.

• Site Visualizer -- Capture any map view or Street View, paint a mask over an area, and describe what you want to build. Gemini 3 Pro Image generates a photorealistic architectural rendering constrained by real zoning regulations -- setbacks, height limits, FAR, and lot dimensions from parcel data.

• Interactive 3D Map -- 8 Milwaukee ESRI data layers (zoning, parcels, TIF districts, opportunity zones, historic districts, and more) rendered with PMTiles for instant performance across 313,000+ features. Includes a measurement tool for calculating area and perimeter.

• 24 Generative UI Cards -- Rich interactive cards for zoning info, parcel details, homes for sale, commercial properties, vacant lots, permit forms, compliance reports, and more -- all rendered directly in the chat.

How we built it

Frontend: Next.js 15 with React 19, RetroUI neobrutalist components, Mapbox GL JS for the map, and CopilotKit for generative UI cards.

Backend: Convex for real-time database, serverless functions, and file storage. All agent logic runs as Convex actions with no client-side API keys.

Gemini 3 Integration (7 models across 10 features):

Key architectural decisions:

• Direct Gemini API calls with manual function-calling orchestration (not a framework wrapper) for full control over tool sequencing and thinking levels
• Model fallback strategy: every API call tries the primary model with 3 retries, then falls back to a secondary model
• Adaptive thinking levels: MINIMAL for fast classification, MEDIUM for vision analysis, HIGH for deep compliance reasoning
• PMTiles on Cloudflare R2 for sub-second map rendering of 313K features

Data sources: Milwaukee ESRI ArcGIS REST services (zoning districts, parcels, TIF, opportunity zones, historic districts, city-owned properties), Mapbox geocoding, and 42 city planning PDFs indexed in Gemini File Search.

Challenges we ran into

Gemini 3 model availability: During peak hackathon hours, Gemini 3 Flash and Pro would intermittently return 429 (rate limit) and 503 (overloaded) errors. We built a robust retry system with exponential backoff and automatic model fallback -- if Flash fails after 3 attempts, the system seamlessly switches to Pro (or vice versa). This resilience pattern is used across all 10 Gemini-powered features.

Document analysis pipeline complexity: Building the classify-enrich-analyze pipeline required careful orchestration. The classification step needs to be fast (MINIMAL thinking), the enrichment step needs to call three external APIs (geocoding, zoning GIS, parcels GIS), and the analysis step needs deep reasoning (HIGH thinking) combined with File Search RAG. Getting these stages to chain reliably with proper error handling at each stage was the hardest engineering challenge.

Map layer event conflicts: Our custom polygon measurement tool initially conflicted with the parcel click handlers across 8 GIS layers. After multiple attempts with the MapboxDraw plugin, we replaced it with a custom, click-based GeoJSON implementation using Turf.js for geodesically accurate area and perimeter calculations.

Convex + Gemini type safety: Convex uses branded types (e.g., Id<"messages">) that don't survive explicit TypeScript annotations. We learned to let Convex infer argument types from validators rather than typing them manually, avoiding subtle runtime errors.

Accomplishments that we're proud of

• 22 agent tools orchestrated in a single conversation turn -- Gemini 3 Flash chains geocoding, GIS queries, parking calculations, RAG searches, and permit recommendations seamlessly through function calling.

• Adaptive thinking levels -- The same Gemini 3 Flash model operates at MINIMAL thinking for instant document classification and HIGH thinking for deep compliance analysis. This demonstrates sophisticated model usage beyond simple prompt-response patterns.

• Zoning-constrained image generation -- Gemini 3 Pro Image generates architectural renderings that respect real dimensional standards (setbacks, height limits, FAR) pulled from Milwaukee's GIS data. This isn't generic image generation -- it's AI architecture grounded in city regulations.

• 42-document RAG corpus -- 5 Gemini File Search stores covering the full Milwaukee zoning code (12 subchapters), 13 area plans, housing policies, and incentive programs. The agent cites specific code sections in its responses.

• Full compliance analysis from a single PDF upload -- Upload a site plan and get back dimensional compliance checks, use compatibility analysis, variance recommendations, and risk assessment -- all cross-referenced against the actual zoning code for the parcel's location.

• 24 generative UI card types -- Every structured response renders as an interactive card, not just text. Homes have images and Street View links. Parcels show zoning on the map. Compliance reports have status indicators for each requirement.

What we learned

• Thinking levels are a game changer. Being able to dial Gemini 3 from MINIMAL to HIGH, thinking on the same model, lets you optimize for speed vs depth within a single pipeline. Classification doesn't need deep reasoning. Compliance analysis does. One model, different modes.

• Function calling scales better than expected. We started with 4 tools and worried about reliability. At 22 tools, Gemini 3 Flash still orchestrates multi-step chains accurately -- geocode, then zoning lookup, then parking calc, then RAG, then permit search -- without getting confused about tool sequencing.

• File Search makes RAG simple. Instead of building a vector database, chunking strategy, and embedding pipeline, Gemini File Search handles it natively. Upload PDFs, create a store, and query. The quality of retrieval from zoning code PDFs exceeded our custom chunking attempts.

• Direct API integration > framework wrappers. We prototyped with Google ADK but moved to direct Gemini API calls for tighter Convex integration and more control over retry logic, thinking levels, and tool orchestration. The extra effort paid off in reliability and flexibility.

• Civic data is more accessible than you think. Milwaukee's ESRI ArcGIS REST services are publicly available with no API key required. The data is excellent -- 313,000+ parcels with zoning, ownership, assessed values, and geometry. The barrier isn't data availability, it's data accessibility.

What's next for MKEdev

• Multi-city expansion -- The architecture is city-agnostic. Any municipality with ESRI GIS services (most US cities) can be supported by swapping data layer URLs and uploading local zoning code PDFs to File Search.

• Permit application generation -- Move beyond recommending permit forms to actually pre-filling them with extracted project data from uploaded documents.

• Multi-agent orchestration -- Specialized agents for feasibility analysis, incentive optimization, and construction cost estimation that collaborate on complex development questions.

• Community engagement -- Let neighborhood associations and alderpersons use MKE.dev to understand development proposals in their districts, with AI-generated impact summaries.

• Mobile-first redesign -- Optimize for field use so developers and inspectors can query zoning and analyze sites from their phones while standing on a lot.

Built With

• clerk
• convex
• gemini3
• hybiscus
• mapbox
• nextjs
• retroui