No internet connection with custom firmware

Is it possible to use custom firmware on the R7800 when using an sip router? I am with bt broadband and am locked into using their home hub 6 which has no modem or bridge mode and is frankly bad.i have tried to put openwrt and dd-wrt on my R7800 but I am unable to connect to the internet. it works fine with stock firmware but qos is pretty useless so really need something better was hoping for openwrt for cake qos but I have got a brick wall at the moment so gone back to stock any help would be great