I had problems this year as well. It was a perplexing problem. Last year I had all outputs of the Pixlight 16 controlling my megatree. This year I'm adding more features using RGB 2811 pixels, so I needed to become more efficient. So I changed the "setup" in Xlights and changed the Advanced String Configurations in Advatek Light Assistant. Nothing seemed to work properly. Like you, all my strings worked using the Pixlight 16 test feature, proving my wiring was correct.
I have 128 channels of LOR, but I'm not using DMX to control LOR. I have the LOR setup over USB port using a serial adapter. So I had to simplify my problem by concentrating on one feature. So I created a new show directory and then created one model in Xlights, but left the Xlight setup alone. Using the Test tab in Xlights I was able to see that the pixels where 129 channels off.
In my Xlight setup, I had the 128 channel of LOR mapped to the first 128 channels of Xlights. So I experimented by moving the LOR channels to the bottom of the Xlight Setup list. So now they are mapped to Xlights channels 6088 to 6215. So now the RGB strings are working correctly. I haven't checked to see if the LOR channels are working.
My recommendation when trouble shooting is to simplify as much as possible to confirm what you know and assume that it's not a hardware problem. I've found that, in my case, all of my problems have been setup and configuration problems, that I have to stumble to find solutions.