A farmer calls and describes a crop problem in Hindi, English, or a mix of both. Instead of answering immediately, the agent asks the questions an agronomist would ask, checks real data, and — when it is not certain — writes up a structured case and books a callback with a human expert. The farmer never has to explain the problem twice.
Voice pipeline
- Deepgram nova-3Speech recognition, language set to multi
One stream carries Hindi and English together, so a farmer can switch language mid-sentence without touching anything
- GPT-5 miniThe reasoning that runs the triage
Decides what to ask next and when to call a tool. Chosen over a smaller model because it actually uses its tools instead of asking the farmer
- MiniMax speech-2.6Hindi voice, hindi_male_1_v2
A Hindi voice reading Devanagari text. An English voice reading romanised Hindi is unintelligible to the person listening
- Agora SD-RTNReal-time audio network, Asia-Pacific region
Carries the call. Pinned to AP so audio does not cross the Pacific and back for a farmer in Maharashtra
Barge-in is on, so the farmer can talk over the agent. Filler words play while a tool runs, because silence on a rural line reads as a dropped call. If the farmer goes quiet, the agent gently checks whether he is still there.
Tools
- get_weatherOpen-Meteo, no API key
Rain and humidity change the diagnosis. The agent looks it up instead of asking a farmer to describe the weather
- search_advisoryCurated knowledge base, 8 conditions across 4 crops
Returns the one question that separates look-alike diseases, so the agent asks instead of guessing
- create_caseStructured record, 12 fields
Turns a conversation into something an agronomist can act on, including how confident the agent actually was
- escalate_to_expertMarks the case and sets urgency
The handover point where a machine stops and a human takes over
- book_expert_callbackGoogle Calendar event and link
Gives the farmer a real time — "tomorrow at ten" — instead of a vague promise that someone will call
Tools are served over the Model Context Protocol. Agora's cloud calls them during the conversation — the requests come inbound to this app, which is the opposite of how the rest of the system flows.
What it will not do
- Never states a pesticide dose, quantity or mixing ratio. That decision belongs to a human, and the knowledge base does not contain doses at all.
- Says out loud how confident it is, and that confidence is passed to the agronomist with the case.
- Escalates whenever it is unsure, the damage is spreading, or the farmer asks for a person.
- Tells anyone who asks that it is a computer assistant working alongside real agronomists.
- On pesticide exposure to a person, it stops advising and tells them to reach a doctor immediately.
Known limits
- The knowledge base covers four crops. Anything outside it is escalated rather than guessed at.
- Speech recognition degrades in heavy field noise and on weak mobile connections.
- The agent is reached from a browser today. A phone line is the obvious next step — the design assumes a farmer on a feature phone with no app and no data — but connecting one was left out of this build rather than half-done.