I built replacements for all of my handheld remote controllers: why and how.
Well, my wife and I were getting old together, and her dexterity with handheld remotes was failing fast. Their shape, odd/inconsistent button configurations and small purpose icons became such a challenge for her that she would just give up. I had already replaced her cell-phone with a large Android pad-phone; big screen with large buttons that laid flat on her table.
So I began consolidating remote controller functionality into big flat menus hosted on ESPs, one for each TV station, den and bedroom. Later I added an adjustable bed ESP controller replacing its hand-held remote as well.
For simplicity, each ESPs supported Alexa voice commands via 'Alexa Routines' although not all remote buttons were so configured.
Then I bought a larger screen TV, LG C5 OLED 77" and Sonos Ultra Arc soundbar for both sight and hearing fade as well as dexterity. This system upgrade required me to also update my iPhone 8 to an iPhone 16 and 'Bam!' my x curl existing menus failed on this new iPhone.
Researching, the issue seems due to a more aggressive iOS screen interface for touch screen controls. Tighter energy management seemed to also play a part by shutting down persistent web socket communications.
I am no developer as this was all for in-home use by an old amateur with time and a need. But all three ESP controlled menus are back up and running on my MacBook Pro and new iPhone. (My wife and iPad are no longer with me, so no testing on an iPad has been done.)
The main problem was tracking a button down event over time to determine the intended interaction type, i.e. keydown, keyheld, keyclick, keyhold, keylong and keyup. 'Keyheld' is a repeating 'keyclick' until the user’s finger is removed from the button. Keyclick, keyhold and keylong are all timed based on keydown and keyup events. This enables each menu button to have multiple actions based on the user's intent.
For example, the powerToggle button’s 'keyclick' can be used for "power on" and it's 'keyhold' or 'keylong' as “power off.”
The menu layout was originally designed for my iPhone 8 with seven rows of three buttons each. With multiple key-types each, the design exceeds all of the original remote controllers.
The design is flexible resizing to maximize use of the available screen. One menu even provides a button to swap menus between the TV and its attached sound bar. Popups are also enabled providing functionality for a scrolling channel selector, as example.
And for the occasional Chrome screen rendering error or other such hangup, a simply twist of the iPhone left or right and back, reloads the menu.
Each ESP host includes a “user interface console” for debugging. It still uses persistent web socket communications. But the menu itself now uses an API to send data to the ESP host.
Finally, the menus and UIC are implemented using vanilla HTML, CSS and JavaScript with no external libraries required. Targeting a Chrome browser device and hosted on an ESP32 was accomplished without Home Assistant nor other Hub and without ESPHome assistance.
It is not my simplistic approach to developing a solution for these human issues as mentioned above. But rather, the results of having voice control for my TVs, sound systems and even my adjustable bed. Even while I’m working on my computer everything is just a spoken phrase or menu click away. All the controls are consistent in looking feel, which is very important to me now since I’m losing my short-term memory as well.
But such as life!