Calibre-Web on RPi4 Connect Cloudflare Tunnel Problems

[MontyJ]

Originally Posted by mariosipad

@MontyJ

If it works, it works! OK.

Do you have a reasonably fixed IPv6 address for your Pi?

Then your clients could access your books over IPv6 (if your router allows pass trough to your Pi.)

I just successfully created a tunnel through cloudflare. It was their 'test' tunnel and does not yet connect to CW, which is of course the whole idea.

No port-forwarding means I have to use some method to get around that, and researching it, many suggested a cloudflare tunnel (free!) would do the trick. And yes, some have suggested it won't work reliably. We shall see.

The CW users who will login remotely are not going to be streaming video, or even music, but just downloading mostly small, 1MB or so files, with an occasional .pdf magazine that could be 200MB or so. So what level of reliability will I need for that kind of activity? Dunno, but will find out.

The other negative comments on using a tunnel on the T-Mobile cellular network is that my public IP address may not really be my public IP address due to some quirk in the way they do double nat and/or IPV6. I haven't a clue if a tunnel will work that way or not, but other comments I have run across says it can be done. And if I can't do in with Cloudflare, then given their network chops, it may not be able to be done.

Anyway, got the test tunnel going, so now going to go through the docs on how to connect it to my web app, CW. Both Cloudflared and CW are in containers, so will have to link them somehow. Who says old dogs can't learn new tricks...

Monty

[MontyJ]

So got the tunnel made (I think, LoL).

Created a config.yml file in the .cloudflared folder, which also contains my tunnel (UUID).json file, and pem.cert file.

Before all this, I have a registered domain, mywebsite.net, all setup in my Cloudflare dashboard. now the last task is connect the tunnel to that website, which is what I think the config.yml does. Here is its simple contents:

url: http://localhost:8083
tunnel: (mytunnel UUID)
credentials-file: /root/.cloudflared/(mytunnel UUID).json

Then it asked me to start routing, as in:

Start routing traffic
Now assign a CNAME record that points traffic to your tunnel subdomain:
If you are connecting an application, route the service to a public hostname:
cloudflared tunnel route dns <UUID or NAME> <hostname>

So I used that format, putting in my tunnel UUID, and mywebsite.net as the hostname, but get errors:

Failed to add route: code: 1003, reason: Failed to create record mywebsite.net with err An A, AAAA, or CNAME record with that host already exists.

So I am obviously misunderstanding something. Yes, I already have a CNAME record, but how to route traffic to the tunnel w/o this step?

Monty

[Quoth]

You need to be having this conversation on a forum specialising in hosting etc. You are self hosting.

This has almost nothing to do with CW (Calibre Web) and the CW has little to do with Calibre support here as it's just using the calibre files.

Also you need a dynamic DNS service as the public IP will change often and because it's cellular/mobile may only activate when your home system connects to the network.

How to see is there NAT in T-Mobile's network?

Use your browser on any site that reports the public IP, like this one: https://www.grc.com/x/ne.dll?bh0bkyd2
Use https://www.grc.com and find "ShieldsUP!" on their website if direct link doesn't fail.

Now see has your Router got stats on IP address. If it's not the same, then there is a NAT on T-Mobile and you can't share to the internet, you'd need hosting and rsync.

You can't use traceroute as it only lists complete routers, not NAT, and usually doesn't show your home public IP but the LAN IP.

My Internet connection on the router statistics is the same as GRC, so there is no NAT. There almost always is on Mobile/Cell, which is what 5G is. Three in Ireland certainly uses NAT.

[Quoth]

Or visit What's My IP
Basically if the IP doesn't match the WAN IP on your router you are behind an ISP NAT and can't share anything.

You can also try traceroute, but it won't directly tell you.

sudo apt install inetutils-traceroute

might install it on a Pi if you don't have it.

If you see 10.x.x.x or 100.x.x.x IP addresses after the 1st LAN IP you are behind a NAT, traceroute www.mobileread.com

If you traceroute your public IP from What's my IP or GRC and you see more than one hop with IPs, then you are behind an ISP NAT and can't share anything. If you see one IP and nothing else, or slow hops with * * * then you are probably public.

If you can forward ports or can't see a WAN IP on the home router, likely yoou are behind an ISP NAT and can't share anything.

If you can see a WAN IP on the Router settings and it's not the same as the public IP from What's My IP or GRC, then you are behind an ISP NAT and can't share anything.

It's extremely rare to be able to share to the Internet with other than xDSL, cable, fibre, leased line or professional radio links. Anything with 5G/4G/3G in the name, or using a router with no port forwarding settings isn't going to work. You then need a hosted website and use a local rsync to keep the files updated. Then CW isn't a good solution unless you have access to install on a virtual server or your own co-located server at a data centre (I've done SW support on a co-located server and had hosted accounts since late 1990s). From 1998 to 2005 I only had dialup. ISDN before that and cable protocol without NAT on a microwave link from 2005 to 2023.

Edit:
https://www.whatsmyip.org/more-info-about-you/ might detect if you have NAT.

[Quoth]

For anyone interested:
https://www.cloudflare.com/en-gb/products/tunnel/

A Cloudflare tunnel is both an anti-DDOS solution and hiding the real location (via geolocate databases) of a server. It does nothing to solve issues of being unable to forward ports from your "server" on the LAN, nor solving issues of being behind a Proxy, NAT or Carrier Grade (CG) NAT of an ISP rather than you router/modem having a public IP.

The only viable solutions to CG-NAT (or other non-public IP on modem/router), or an ISP that won't let you forward ports, are either hosting or a co-located server.

[chaley]

Originally Posted by Quoth

The only viable solutions to CG-NAT (or other non-public IP on modem/router), or an ISP that won't let you forward ports, are either hosting or a co-located server.

I gave up self-hosting years ago. These days a small shared VPS costs $5/month (plus tax if charged) at Linode. Having someone else deal with networking access, hardware maintenenance, 24/7 access, and system updates is worth that for me. No tunnels, no dynamic DNS, fixed IP, fast networking, no security holes into my home network, and so on. Linode provides DNS so that problem goes away as well. Of course the server is operating on a copy so I can't maintain my calibre library using the content server, but I wouldn't do that anyway. I don't use annotations so I'm not concerned about syncing them.

(Full disclosure: I pay $7.40 + VAT because I have them do backups.)

[mariosipad]

@MontyJ

Did you have your Clouflare tunnel working?

I have mine working, but I am using regular cable network and not mobile G4/G5.

[Quoth]

Originally Posted by mariosipad

@MontyJ

Did you have your Clouflare tunnel working?

I have mine working, but I am using regular cable network and not mobile G4/G5.

Cable doesn't usually have an ISP/Carrier NAT, Mobile always does.

[mariosipad]

@Quoth

Please correct me if I'm wrong, but I think there is a fundamental flaw in your understanding of how cloudflared tunnels work.

1 - you build a tunnel from inside the LAN to cloudflare (no NAT or port forwarding needed.
2 - you connect to cloudflare via a domain name where (purchased from cloud flare or any other sellers) the nameservers are set to cloudflare.

So from a browser you connect to cloudflare and cloudflare proxies your connect to your home network trough the tunnel.

Is it safe? I think not 100%, cloudflare knows what you are doing. But no need to open ports on our router. And you still need to protect your sites with passwords when appropriate. Anyone can who can guess your domainname can access your webserver and all that's on it, so lock down that what you want to keep private.

In the end it alcoves down to: "do I trust cloudflare (enough)?"

[Quoth]

That will be deemed illegal misuse by most ISPs and I doubt it will work well on Mobile. I used to work for an ISP.

Hosting with rsync will work on anything. Cloudflare Tunnel is a stupid solution for self hosting and it's designed to mitigate DDOS and hide where you really are. For the tunnel to work without port forwarding across NAT it's essentially creating a virtual site on a Cloudflare server and using a sync process on your LAN. There is no other way for it to work.

It's breaking T&C of any domestic ISP contract I've ever seen, especially 3G
/4G/5G services.

The only thing I ever self hosted was a VPN server on my LAN and its port was forwarded to 80 and 433. Then using a VPN client at a public WiFi I could safely access my email via my home internet. My two sons used it also at university as only the University email worked and all ports except for web browsing were blocked. Since that was personal use by the people living here it didn't contravene T&C. Our home internet was 8 Mbps down / 1 Mbps up (typcally 7.4 Mbps & 0.95 Mbps), so the VPN gave 1 Mbps download and 8 Mbps upload to the home LAN. It was DOCSIS (cable protocol) over a 13km microwave link. I shut it down in 2010 when I stopped travelling.
Now we are on fibre and it's 500 Mbps down and 50 Mbps up. Speedtest just now gives download 554.18 and upload Mbps 55.34, but the T&C still forbid self hosting, which other than a secured VPN for you own private use of email and banking out on a public Wifi is a mug's game. Public WiFi can have MiM attacks on HTTPS.

[mariosipad]

Originally Posted by Quoth

That will be deemed illegal misuse by most ISPs and I doubt it will work well on Mobile. I used to work for an ISP.

It could be "misuse" depending on your contract with an ISP. However it probably only is if you use a lot of bandwidth this way.
My ISP could not care less as long as use is not excessive as in using 100% bandwidth 24/7.

If it works on mobile (as a server) I do not know. Maybe @MontyJ can confirm it works or not?

Originally Posted by Quoth

Hosting with rsync will work on anything. Cloudflare Tunnel is a stupid solution for self hosting and it's designed to mitigate DDOS and hide where you really are. For the tunnel to work without port forwarding across NAT it's essentially creating a virtual site on a Cloudflare server and using a sync process on your LAN. There is no other way for it to work.

I do not disagree with you here. However it might be the only way open to someone who does not or cannot open ports in his/her router and use port forwarding.
Also hosting books is better not done on a public server. The legality of hosting (even storing) books night be questionable (even illegal) depending on where you live. So hosting on your own server protected by passwords and SSL is the only option.

Originally Posted by Quoth

It's breaking T&C of any domestic ISP contract I've ever seen, especially 3G
/4G/5G services.

Possible even probable.

Originally Posted by Quoth

The only thing I ever self hosted was a VPN server on my LAN and its port was forwarded to 80 and 433. Then using a VPN client at a public WiFi I could safely access my email via my home internet. My two sons used it also at university as only the University email worked and all ports except for web browsing were blocked. Since that was personal use by the people living here it didn't contravene T&C. Our home internet was 8 Mbps down / 1 Mbps up (typcally 7.4 Mbps & 0.95 Mbps), so the VPN gave 1 Mbps download and 8 Mbps upload to the home LAN. It was DOCSIS (cable protocol) over a 13km microwave link. I shut it down in 2010 when I stopped travelling.
Now we are on fibre and it's 500 Mbps down and 50 Mbps up. Speedtest just now gives download 554.18 and upload Mbps 55.34, but the T&C still forbid self hosting, which other than a secured VPN for you own private use of email and banking out on a public Wifi is a mug's game. Public WiFi can have MiM attacks on HTTPS.

Again, it depends on your ISP and where you live. My IPS even helps with howto's for portforwarding. Running my own IPv4 and IPv6 server(s) for gaming, websites, VPN's, etc. is allowed.
I sometimes forget how restrictive internet can be in some parts of the world or with some IPS's.

.

[Quoth]

So hosting on your own server protected by passwords and SSL is the only option.

Nonsense. That's worse than a hosted account.
If it's not legal on a hosting service it's not legal self hosted. Hosting can have private access, then copyright material is possible. It has to be genuinely your own use. It's copyright violation to give the access to anyone else no matter if on your own server or hosted account. It's delusional to think otherwisre.

Open to the public (i.e. other than personal) copyright material will get you banned from ISP, CloudFlare or Hosting no matter how it's shared, even if there is SSL and a password.

[Quoth]

Originally Posted by mariosipad

Again, it depends on your ISP and where you live. My IPS even helps with howto's for portforwarding. Running my own IPv4 and IPv6 server(s) for gaming, websites, VPN's, etc. is allowed.
I sometimes forget how restrictive internet can be in some parts of the world or with some IPS's.

All real broadband ISPs will help with portforwarding, gaming, VPNs etc.
Almost none allow a website, that's madness even just from a security viewpoint. My hosting account with loads of websites, near unlimited everything (fair use T&C) and loads of domains and email cost is a fraction of my fibre broadband cost and much faster. They look after security, assuming my CMS and passwords are sane.

Yes, they care most about traffic, secondly illegal content, thirdly copyright violations.

Home hosting / self hosting makes no difference to any copyright or other issues with content.

[pazos]

Originally Posted by Quoth

That will be deemed illegal misuse by most ISPs and I doubt it will work well on Mobile. I used to work for an ISP.

Hosting with rsync will work on anything. Cloudflare Tunnel is a stupid solution for self hosting and it's designed to mitigate DDOS and hide where you really are. For the tunnel to work without port forwarding across NAT it's essentially creating a virtual site on a Cloudflare server and using a sync process on your LAN. There is no other way for it to work.

It's breaking T&C of any domestic ISP contract I've ever seen, especially 3G
/4G/5G services.

These are a series of bold statements full of BS and without any backing data to prove them true.

I would suggest you to learn a bit before talking. No matter whatever work you did in the past.

So, please explain how a GRE tunnel is different from an SSH tunnel, a VPN tunnel or any other tunnel that encapsulates a packet within another packet. Is you ISP agaisnt encapsulation? How about TLS?

Cloudflared tunnels are a great way of "self" hosting, most likely the easier to implement behind a CGNAT. ISPs have nothing to do here. If you can break somebody's ToS it will be Cloudflare's, which is the one providing the service.

So, nope. Tunnels are legal, fine and used everywhere. You'll need to trust the remote endpoint. That's it.

[Quoth]

Originally Posted by pazos

So, please explain how a GRE tunnel is different from an SSH tunnel, a VPN tunnel or any other tunnel that encapsulates a packet within another packet. Is you ISP agaisnt encapsulation? How about TLS?

Irrelevant. ISPs don't care about encapsulation*. I wrote "they care most about traffic, secondly illegal content, thirdly copyright violations."

Originally Posted by pazos

Cloudflared tunnels are a great way of "self" hosting, most likely the easier to implement behind a CGNAT. ISPs have nothing to do here. If you can break somebody's ToS it will be Cloudflare's, which is the one providing the service.

So, nope. Tunnels are legal, fine and used everywhere. You'll need to trust the remote endpoint. That's it.

It's not whasover about legality of tunnels.

ISPs care most about traffic, secondly illegal content, thirdly copyright violations.

Many specifically forbid hosting, i.e. running a website from home.

It's irrelevant how you host as regards copyright content (hom, co-locate or hosting service).

It's irrelevant what's between your home server and internet users.

Also it doesn't matter what the peak speed is, cell/mobile is connection on demand and may not connect. If you are accessing your home from the internet, rather than vice versa, you really need broadband.

@Pazos you are totally missing the point!

1) Cloudflare has no effect on legality of the content.
2) Mobile / Cell (3G/4G/5G etc) is rubbish for inward traffic or reliability. You need real broadband.

Go read Cloudflares adverts & FAQ for tunnels. It's not about legality of Cloudflare used for a tunnel. It's pointless compared to hosting [a web site], which is a fraction of the cost of proper broadband.

EDIT
[* Unless it's two way geosyncronous satellite in which case only the providers VPN works. Or if you live in a totalitarian country you better make your encapsulation look like something else.]