Custom dialog box in Automator? I would like to create a service that performs a set of actions based on several parameters
A custom dialog box with multiple radio buttons would be a great way to capture these.
My experience with AppleScript so far only gives me very basic prompts
How can I use a custom dialog box in Automator?
Maybe this will be of use to you.

Dialog Toolkit and Dialog Toolkit Plus are ASObjC-based script libraries for showing enhanced dialogs. Add multiple text entry fields, checkboxes, popup menus, radio buttons, path controls, extra buttons and rules to dialogs. 

Just read the text files included in the bundle to help you get started. Playing around in Script Editor.app, with the sample scripts included, should help you understand the process.

Here are a few examples created with this scripting addition in Script Editor