How to remotely control *any* iOS app!

edited February 11 in App Tips and Tricks

Hi folks,

i think it's about time to start a mini tutorial about how to control any iOS app on iOS versions >=iOS 9.0 because from many questions asked repeatedly, it seems like most iOS musicians are not yet aware of the built-in remote control features of recent iOS versions.
You might want to save this for your own reference so when you need it, you can follow it step-by-step.

Until iOS 8.x, Accessibility options were somewhat restricted. It was possible to control an on-screen cursor and select items, it was also possible to record custom gestures, but freely assigning incoming control messages to any gesture is a feature that was (finally!) added with iOS 9.0.

What's the deal?
Many hardware controllers connected via Bluetooth, USB or MIDI can be used to trigger any touch event or touch sequence that you can freely define.
These can be keys on a Bluetooth keyboard, a Bluetooth foot pedal, MIDI notes on a controller keyboard, buttons on a compatible bluetooth game controller etc.

The principle of remote-controlling any app is to:
1. Add and name a "Switch" for each control on your controller hardware
2. Create a new recipe for the app you're about to remotely control
3. For each switch, record a custom gesture for this app
4. Give the gestures appropriate names so you remember what they do.

Switches
You start by connecting your controller hardware to your iDevice.
While USB devices are usually plug-and-play, Bluetooth devices have to be paired first.
Then go to Settings > Accessibility > Switch Control and make sure you leave "Switch Control" = OFF for now.
Tap on "Switches", scroll down and tap "Add New Switch", then Source=External.
Now toggle the switch/key/button you want to learn and give it a recognisable name. I usually call it _, like "Blueboard_4" for example.
Hit "Save".
iOS will ask you to choose a switch action now. Choose "SYSTEM" > "Tap"
Go back to the "Switches" page.
Your new switch and name must be listed here.
Do this for every switch on the controller that you want to assign a function to.

Recipes
Now that your iDevice knows a few controller events by name, you are going to create a set of gestures for the app you want to control.

Go back to Settings > Accessibility > Switch Control and tap on "Recipes".
Tap on "Create New Recipe...", give it a name that tells you what it does in the app under control, leave the "Timeout" switch off and tap on "Assign a Switch...".
From the list of your switches, select the remote controller switch you want to assign an action to.
Now is a good time to open the app you want to control and remember the control and gesture positions that have to be tapped/swiped/dragged to activate the function you need. Sometimes, tap positions can be remembered by putting your hand on the iDevice and hold your index finger in-place with the help of thumb and middle finger, but a much more precise option is to use a CD marker pen (formerly overhead slide pen) that works when writing on glass or protective film.
The way I do it: I'm using a fine pen and in the middle of each of the app's UI elements to control, I'm writing small consecutive numbers in the order I want to control them by buttons on the remote controller device.

Now switch back to the iOS Settings app.
Under "New Recipe" > "Assign a Switch..." > "Switches" > {Button-on-your-remote}!
Select "Actions" > "Custom Gesture", tap "Hide Controls" on the bottom and record your gesture, which can be as simple as a short tap on the correct position.
A few notes about recording gestures:
1. Recording starts as soon as your finger touches the screen. Because a gesture can consist of multiple taps, swipes, drags etc., recording will stop automatically 10 seconds after your last finger lift-off from the screen.
2. Make sure you tap position is precise enough to tap on the correct UI element on the app under control. Some developers don't add a "tolerance range" around the UI elements so you really don't want to miss the correct position.
3. Your gesture will be replayed with the same speed you've recorded it. A long tap might be interpreted as a tap-hold, moving your finger while releasing it from the screen might be interpreted as a swipe or unintentional parameter change.
4. Your gesture will not be aware of the current status of the UI element. Example: You can record a "scroll bar" gesture to scroll up or down in a list, but it won't work well except when the scroll handle is at the correct position by coincidence.
5. However, gestures requiring multiple taps like opening a menu, selecting a sub menu and then selecting an item, are perfectly possible if menu positions don't change depending on context.
6. You might want to practice entering your gestures to be both fast and precise!

Repeat this procedure for each switch until all controls from your remote are mapped the way you want them to work.

Finally, make sure you select the recipe you just created under Recipes > "Launch Recipe" so when you activate Switch Control, the correct set of remote-controlled gestures is used for the app under control.
Then, under Accessibility > Switch Control, set "Switch Control" to ON.

Switch to your new controllable app and go ahead loop-recording, track arming, chords playing, sampling, selecting, menu diving, complicated-sequence-triggering etc.

Just an idea: By attaching a Bluetooth computer keyboard and recording gestures for each key, you can easily have more than 70 actions - a different one on each key. Let's talk Beatmaker 3 workflows :D

Good luck and have fun liberating more time for making music!

Edit 1: Some additional settings under Settings > Accessibility > Switch Control:
Scanning Style = Manual
Tap Behaviour = Always Tap
Focused Item After Tap = Current
Saved Gestures = None
Dunno if they make any difference, just added these for completeness' sake.

Edit 2: iOS 11 is apparently required to use MIDI Notes for remote control.
Bluetooth keyboards also work under earlier versions.
There seems to be a bug in iOS 12.x that prevents MIDI switches from working at all, so stay on iOS 11.x if you can!
The bug has already been reported to Apple today.

«13

Comments

  • :open_mouth:

    that sounds pretty great, looking forward to try it, thanks for tutorial !!!

    :+1: :+1: :+1:

  • Interesting. A little Bluetooth number pad and your up and running!

  • @Dr_M said:
    Interesting. A little Bluetooth number pad and your up and running!

    Yes!!

  • edited February 10

    Thanks for the tutorial! I didn’t know you could do this since ios 9. If I know anything about myself, it’s my passion for controllerism.
    Considering the advancements in Midi that have been attained for iOS, this might be excellent for drawing, photography, and video editing.

  • edited February 10

    @rs2000

    Which device are you using to control?

    My NanoKontrol Studio is showing as connected.

    However when I get to the..

    Tap on "Switches", scroll down and tap "Add New Switch", then Source=External.
    Now toggle the switch/key/button you want to learn and give it a recognisable name. I usually call it _, like "Blueboard_4" for example.

    Nothing happens or is recognised.

  • edited February 10

    I’m able to connect BlueBoard and create buttons and recipes, but I’m not able to make all of this works.

    Super interesting and thanks for sharing @rs2000, do you think you could make a short video showing the basic process to complement this tutorial?

  • When i go to ‘create new recipe’ and name it there is no ‘assign the switch’ button, just ‘timeout’. Not sure if ‘assign the switch’ should be on that same page?

  • Holy Hax, Batman!

  • @SpookyZoo said:
    @rs2000

    Which device are you using to control?

    My NanoKontrol Studio is showing as connected.

    However when I get to the..

    Tap on "Switches", scroll down and tap "Add New Switch", then Source=External.
    Now toggle the switch/key/button you want to learn and give it a recognisable name. I usually call it _, like "Blueboard_4" for example.

    Nothing happens or is recognised.

    Make sure you have assigned MIDI notes to the keys or buttons (using KORG's KONTROL editor). I haven't had any luck with MIDI CC messages yet.

  • @supadom said:
    When i go to ‘create new recipe’ and name it there is no ‘assign the switch’ button, just ‘timeout’. Not sure if ‘assign the switch’ should be on that same page?

    Yes it should. Are you sure you have recorded a number of external switches first?

  • edited February 11

    Ok, so for the tl;dr are you saying that iOS system settings has an option to make custom touch sequences that are triggered by MIDI ??????? And this is completely accessible in iOS settings without any third-party app??

    If so, this is just mind-blowing. Seriously, I can’t even wrap my mind around the possibilities.

  • @Hmtx said:
    Ok, so for the tl;dr are you saying that iOS system settings has an option to make custom touch sequences that are triggered by MIDI ??????? And this is completely accessible in iOS settings without any third-party app??

    If so, this is just mind-blowing. Seriously, I can’t even wrap my mind around the possibilities.

    Absolutely. I find it just as mind-blowing to enable remote control for apps that don't support it natively.

  • How does the iOS system handle midi input? Does it have to be hardwired over USB or network midi? Can it be virtual midi? I’m so intrigued by this. Thanks @rs2000

  • edited February 11

    @Hmtx said:
    How does the iOS system handle midi input? Does it have to be hardwired over USB or network midi? Can it be virtual midi? I’m so intrigued by this. Thanks @rs2000

    I've tried USB MIDI, Bluetooth MIDI, Bluetooth HID (a BT wireless keyboard) and a wireless keyboard with a proprietary (non-BT) receiver dongle attached via CCK2.
    Make sure you're sending MIDI Notes as triggers. MIDI CC didn't work for me.
    The IK Blueboard has a mode that sends MIDI Notes, you might have to enable it.

  • @rs2000 said:

    @supadom said:
    When i go to ‘create new recipe’ and name it there is no ‘assign the switch’ button, just ‘timeout’. Not sure if ‘assign the switch’ should be on that same page?

    Yes it should. Are you sure you have recorded a number of external switches first?

    I've recorded one external switch on my bt keyboard.

  • edited February 11

    @supadom said:

    @rs2000 said:

    @supadom said:
    When i go to ‘create new recipe’ and name it there is no ‘assign the switch’ button, just ‘timeout’. Not sure if ‘assign the switch’ should be on that same page?

    Yes it should. Are you sure you have recorded a number of external switches first?

    I've recorded one external switch on my bt keyboard.

    Strange. This is how the dialog looks here (iPad, iOS 11.4.1):

  • edited February 11

    Thanks for the tutorial. :)
    It's interesting but I'm not sure of it's use yet, but would like to see how you/others are using it...
    Since it's designed for use when you don't have access to the screen either from disability or lack of access. If you do have access then this is always preferred. I always get these annoying scanning blue boxes and there seems to be no way to disable this. So whenever I've looked into this before I've thought it could be useful but never actually did anything with it worthwhile...

    +1 for a demo video of any useful recipes :)

  • edited February 11

    @Carnbot said:
    Thanks for the tutorial. :)
    It's interesting but I'm not sure of it's use yet, but would like to see how you/others are using it...
    Since it's designed for use when you don't have access to the screen either from disability or lack of access. If you do have access then this is always preferred. I always get these annoying scanning blue boxes and there seems to be no way to disable this. So whenever I've looked into this before I've thought it could be useful but never actually did anything with it worthwhile...

    +1 for a demo video of any useful recipes :)

    Of course you can simply use the IK Blueboard foot pedal, but here's how I'm using it:
    After finding out how to use this feature, I grabbed me a cheapo Bluetooth Number Pad and an old Ibanez IFC60 foot pedal. With a bit of soldering, I've connected its 6 foot switches to keys 1..6 of the key pad so i can remote-control Loopy HD and Genome MIDI.
    One tipp: When buying a Bluetooth keyboard for such a project, make sure it's really Bluetooth and not just a wireless keyboard, and also make sure you don't have to enter a PIN code when pairing! Most keyboards require it for security reasons, and you really don't want that in a foot pedal :D

    PS: Sorry I don't record videos as I personally prefer written step-by-step instructions.
    If you find any mistakes or omissions in my description, please PM :)

  • edited February 11

    .

  • edited February 11

    @rs2000 said:

    I've tried USB MIDI, Bluetooth MIDI, Bluetooth HID (a BT wireless keyboard) and a wireless keyboard with a proprietary (non-BT) receiver dongle attached via CCK2.

    Are you saying that a wireless keyboard does work? That would be interesting

  • @Hmtx said:

    @rs2000 said:

    I've tried USB MIDI, Bluetooth MIDI, Bluetooth HID (a BT wireless keyboard) and a wireless keyboard with a proprietary (non-BT) receiver dongle attached via CCK2.

    Are you saying that a wireless keyboard does work? That would be interesting

    Yep, as long as the supplied USB dongle acts as a class compliant HID device.
    Mine works and I guess most wireless keyboards on the market do as well.
    Bluetooth has the advantage that you don't need a CCK and the USB port remains free for your Audio/MIDI interface.

  • edited February 11

    @rs2000 I’ve done a short speechless video to show you what I do. I think I’ve followed properly your instructions but that don’t works. Please let me know what I’m doing wrong. This is with BlueBoard and Loopy which is easy for gesture test:

  • edited February 11

    @Janosax said:
    @rs2000 I’ve done a short speechless video to show you what I do. I think I’ve followed properly your instructions but that don’t works. Please let me know what I’m doing wrong. This is with BlueBoard and Loopy which is easy for gesture test:

    Yes, that's exactly how I did it.
    I have just tried it on the iPhone (iOS12) and you're right, MIDI doesn't work there.
    The Bluetooth keyboards do work however.

    Either it's an issue with the iPhone or something is broken in iOS 12.

    Can anyone with iOS 12.x on the iPad try if control by MIDI messages works please?

  • Yeah, that explains why it wasn't working for me either, ipad 12.9 pro2.

  • Too bad, a nice use would have been to launch Gadget scenes by midi... It could be interesting to know which iOS versions are not broken.

  • @Janosax said:
    Too bad, a nice use would have been to launch Gadget scenes by midi... It could be interesting to know which iOS versions are not broken.

    iOS 11.4.1 definitely works.

  • @rs2000 said:

    @Janosax said:
    Too bad, a nice use would have been to launch Gadget scenes by midi... It could be interesting to know which iOS versions are not broken.

    iOS 11.4.1 definitely works.

    This is worth a bug report @ Apple me thinks.

  • edited February 11

    @rs2000 said:

    @rs2000 said:

    @Janosax said:
    Too bad, a nice use would have been to launch Gadget scenes by midi... It could be interesting to know which iOS versions are not broken.

    iOS 11.4.1 definitely works.

    This is worth a bug report @ Apple me thinks.

    Yes that’s worth it for sure. The issue with Apple bug report is their lack of communication. I’ve reported to them the GarageBand undo bug some months ago, they asked me for some song files and then no news. It has been fixed lately, so that’s cool, but the user need to test from time to time after an update to know it. However, I will look to report this one too.

    Just a question: as you assign a recipe to « launch recipe », does that mean only this recipe will works? Or you can have several recipes?

  • @rs2000 said:

    @rs2000 said:

    @Janosax said:
    Too bad, a nice use would have been to launch Gadget scenes by midi... It could be interesting to know which iOS versions are not broken.

    iOS 11.4.1 definitely works.

    This is worth a bug report @ Apple me thinks.

    @rs2000 Thanks for this - another very cool technique to play with!

    I'm on iOS 12.1 and I can get this to work with a Bluetooth keyboard, but not a MIDI keyboard. I can assign the Switches, but I can't trigger the Recipes; gonna keep playing with this.

    The idea is especially appealing for apps that have gestural controls, but no motion recording (SpaceCraft, Shoom, etc.) The end result isn't as clean as true motion recording as you have to pantomime the gestures on the blank recording screen (I wish you could load a screenshot...) and sometimes apps don't quite respond correctly. Still this is very fun and very cool.

  • @Janosax said:

    @rs2000 said:

    @rs2000 said:

    @Janosax said:
    Too bad, a nice use would have been to launch Gadget scenes by midi... It could be interesting to know which iOS versions are not broken.

    iOS 11.4.1 definitely works.

    This is worth a bug report @ Apple me thinks.

    Yes that’s worth it for sure. The issue with Apple bug report is their lack of communication. I’ve reported to them the GarageBand undo bug some months ago, they asked me for some song files and then no news. It has been fixed lately, so that’s cool, but the user need to test from time to time after an update to know it. However, I will look to report this one too.

    Just a question: as you assign a recipe to « launch recipe », does that mean only this recipe will works? Or you can have several recipes?

    Best as I can tell you only have that recipe. However, you can assign multiple switches and gestures to a single recipe....

Sign In or Register to comment.