Using Python to Code by Voice

> The second thing we need to figure out is how we can compress voice input to make it faster to transmit. What’s the voice equivalent of a thumbs-up or a keyboard shortcut? Can I prompt Claude faster with simple sounds and whistles? This reminds me of the amazing 2013 video of Travis Rudd coding python by voice: ▶ this video The number of times in the last few years I've wanted that level of "verbal hotkeys"... The latencies of many coding llms are still a little bit too low to allow for my ideal level of flow (though admittedly I haven't tried one's hosted on services like groq), but I can clearly envision a time when I'm issuing tight commands to a coder model that's chatting with me and watching my program evolve on screen in real time. On a somewhat related note to conversational interfaces, the other day I wanted to study some first aid stuff - used Gemini to read the whole textbook and generate Anki flash cards, then copied and pasted the flashcards directly into chat GPT voice mode and had it quiz me. That was probably the most miraculous experience of voice interface I've had in a long time - I could do chores while being constantly quizzed on what I wanted to learn, and anytime I had a question or comment I could just ask it to explain or expound on a term or tangent.

In 2017 I spent a while messing around and creating a system to code and control my computer via voice. I was experiencing RSI pain at the time, and thought I should be proactive and have a strategy where I could still work and use my computer in case it kept getting worse and it became an impedance to create such a tool. I tried every voice to text I could find, and unfortunately for me the only acceptable one in terms of quality was Dragon Naturally Speaking, which was commercial and Windows only (I use Linux). I decided to build a virtual machine running Windows XP which ran the voice -> text translation, and then run a local server on the Linux side which would receive packets of text from the virtual machine. It was then a matter of parsing the string for language primitives, as you'd need a custom alphabet of keywords to do certain actions like type any given key combination, and inventing your own primitives for this reduces ambiguity (voice detection is only so accurate and the use case here means it's going to be less accurate than usual since you are not speaking in expected english, plus you want everything to be single syllable). The process of building a dictionary of primitives and shorts was very much akin to what court reporters / Stenographers do to type fast, and was also probably related to my RSI given that I started my career out as a Stenographer. Something I regret in retrospect. In terms of voice coding, things really have gotten so much better since then where we now have amazing free and open source options for text to speech, and we've also seen a proliferation of apps used to code via voice. I'm partial to Talon, though I don't do any voice coding today. link Github also just announced a voice to code copilot type thing, and at this point given the advances we're seeing in AI I'm sure I'll be okay if my RSI gets bad. This video was one of the things I watched and helped me in building the system, ▶ this video I'm also building a video game, and plan on building many more. I'm writing it in a monorepo where I have a common shared foundation, and then apps using and building on that foundation. I believe in dogfooding my code, and have built a bunch of things with it towards that end The thing I'm happiest with and use the most is a small and simple music player. I never could find a replacement Foobar2000, so I wrote my own. It runs nearly 24/7 on my PC's. I've also built a breathing app after discovering that breathing exercises were like magic in terms of improving mood and reducing blood pressure. The one I built was modeled after link and I mainly built it because it was trivial to do and Firefox kept putting the web tab to sleep. If you have high blood pressure, I 100% recommend exploring different breathing exercises. I've also built two different GUI wrappers around image generators. The first app was built around VQGan+Clip back before Stable Diffusion, and it supported swapping the backends to change generators. I built it as a web app with Svelte, and it let me explore the images and auto-generate based on a theme or with a given sentence structure where parts of the sentence could be sampled from a pool. The second one was much the same, but it was built with my monorepo, it was built around Stable Diffusion, and I added an image-to-image component where I have a simple UI to draw on the input image through the app. The usefulness of this project is near 0 as there are better open source versions out there. I also built a static website generator in Ruby for my personal website. I've since soured on Ruby though, and my website is no longer online. There are other things but I'll leave it there because this is already too long.

There was a great talk from 2013 about coding with voice commands: ▶ this video It's by a developer who developed RSI and had to find another way to write code. He uses a combination of Dragon and custom Python scripts to control Emacs. The fascinating bit for me was the language he created around text navigation and manipulation. Lots of custom short words to optimise the amount of speaking he actually had to do. Really worth a watch for anyone interested in this. If you want a quick demo, this part of the video is fairly representative: ▶ this video

I have been suffering from RSI in both arms for a decade now. It started after coding 10+ hour days working at a game company. The pain ebbs and flows with usage, but the inflammation and tightness has never disappeared completely. Over the years, I've met a number of people who also also suffer from RSI in various forms. Fortunately, the majority of people I talk to suffer for some period of time, but do eventually recover, and are able to continue on with their lives having to be carful to avoid future flareups. A handful though, including myself, seem to fall into the "for life" category. I've tried everything you can think of to varying results: * 15+ doctors in various specialties * Physical therapy * Massage therapists * Chiropractors * Acupuncture * Most every ergonomic keyboard and pointing device on the market * Various desk/chair/keyboard tray ergonomic adjustments * Addressing TMS (See books by Dr. Sarno) * Taking a year off of work (multiple times) * NSAIDs This is resulting in me spending a lot of time looking into other career potabilities, but I keep ending up back writing code to some extent. Voice coding is literally the first thing that comes to everyone's mind when you tell them you have arm pain, but it's a dead end in my mind. I tried to follow in the footsteps of Tavis Rudd, but I just ended up with voice strain from talking too much. You might have better luck though: ▶ this video At one point in my life, I had an intern who knew how to code type for me. No voice rec system will ever be as good as a human who knows how to code... and even dictating this way was slow and frustrating, though at least my voice wasn't strained. Someone else here mentioned TMS. A good friend of mine who was suffering for months with arm pain was greatly helped by looking into and addressing it. I recommend Dr Sarno's book "The Mindbody Prescription". Even though it wasn't a cure for me personally, I did find it helpful to mitigate some of the pain. As far as creating a community goes, I've never found reading about others suffering to be any help or comfort. If anything I've found it better to not dwell on it and just move on with life. OTOH, a resource that offered suggestions and opportunities on how to make the most of the capabilities that are still available might be nice. Knowing there's still a lot that one can accomplished, even if you can't code for 8+ hours a day, 5 days a week.

Nothing out of the box as far as I know. You'll have to DIY. Have a look at CMUSphinx/Pocketsphinx [1]. I wrote a comment about training it for command recognition in a previous discussion[2]. It supports BNF grammar based training too [3], so I've a vague idea that it may be possible to use your programmming language's BNF to make it recognize language tokens. I haven't tried this out though. Either way, be prepared to spend some time on the training. That's the hardest part with sphinx. Also, have you seen this talk for doing it on Unix/Mac [4]? He does use natlink and dragonfly, but perhaps some concepts can be transferred to sphinx too? [1]: link [2]: link [3]: link [4]: ▶ this video

There was an interesting talk at PyCon 2013 about programming using only your voice: ▶ this video Might be worth looking into for your case. It's worth mentioning that the guy did this for himself due to severe RSI in the hands, but there was another talk on using a stenographic keyboard to program and I remember this guy stood up and said he wished he'd known about it earlier as it's another approach for people who suffer from RSI since there's less typing.

“As someone who easily spends more than 60 hours a week at the keyboard gaming, coding, working, and writing, I am no stranger to wrist pain. Particularly the part where my thumb and wrist meet on my right hand.” Patient: “Doctor, doctor, it hurts when I do this.” Doctor: “Then don’t do that!” Perhaps it would be a good idea to reduce typing in general? We’ve got voice input that’s getting much better. We had this a decade ago: ▶ this video Some sort of gesture based input? AR/VR has helped improve this? Hopefully a combination of keyboard, voice, and gestures could reduce RSI issues.

“I think there is a world market for maybe five computers.” - Thomas Watson I bet if we use our imaginations, we’ll think of a lot of places were using voice to code could come in handy. Personally, I’ve been waiting for it for a few decades. The creator of TCL has RSI and has been using voice since the late 1990’s link Thought we were really close 10 years ago when Tavis Rudd developed a system: ▶ this video GitHub seems to be more high-level. It figures out the syntax and what you actually want to write. This would help if you barely knew the language. Time to learn Rust or Scala with a little help from machine learning.