I'm told that GitHub has asserted to us that moving to this model means we would not be exposed to github.com outages. It's not at feature parity with github.com though.
“GitHub Enterprise Cloud with data residency” is hosted on separate infrastructure and dedicated subdomains under *.ghe.com. It’s been around since November 2024z
It’s not the same thing as GitHub Enterprise Cloud hosted on the shared global network on github.com.
That is a different and later product with a confusingly similar name.
vinnymac 18 minutes ago [-]
Just to be clear, I am on Github Enterprise, and am also experiencing this disruption both privately and publicly on every org and project I have access to.
VCFundedGenYer 39 minutes ago [-]
That's what Azure DevOps is supposed to do, but for some reason GitHub has a redundant enterprise division.
weli 1 hours ago [-]
That's what I don't understand. They could mitigate their name so much if they just split free/paid/enterprise. It's already shown that enterprise is much more estable and is largely unaffected from service disruptions. Why don't they go one more layer? For sure it's worth the extra complexity.
lbriner 37 minutes ago [-]
There is no such thing as "just split" there is 20+ years of legacy decisions and even if the split is relatively clean it is still probably 1 years work for 200 people for maybe a marginal improvement.
The real money is going to go towards, "make this all more reliable".
gaigalas 56 minutes ago [-]
Depending on the cause of the current issues, that move would likely cause more harm to paid services than good.
Their last postmortem made clear that their challenges are operational. Scale puts pressure on operation, but it's not what blocks them from keeping up.
Doubling the operation doubles the operational challenges.
rethab 58 minutes ago [-]
surely if they did that everybody would complain how github "lost its touch with open source since they now prioritize paid services"
john_strinlai 1 hours ago [-]
enterprise is mostly separate, is it not? uptimes are significantly more reasonable on the enterprise status pages
saxonww 55 minutes ago [-]
We are in GHEC right now and GitHub Actions is not working. It's been down every time githubstatus.com says it's down.
mh- 49 minutes ago [-]
Same for us, I'm not even sure what product that other "Enterprise" status page refers to..
tomw1808 54 minutes ago [-]
Things can go wrong, but really, its been a lot and we're normalizing that to an unhealthy degree...
I wonder if it was down that much, if users would get credits the way we pay when we use the services - its kind of ridiculous for a critical service to be down that much and all we do is "ah okay, its just github". Like, as if that was normal to be down that much...
zaik 38 minutes ago [-]
I think the authors of SMTP had a healthy attitude to server uptimes:
Retries continue until the message is transmitted or the sender gives
up; the give-up time generally needs to be at least 4-5 days.
Common Name (CN) www.dayswithoutgithubincident.com
Organization (O) <Not Part Of Certificate>
Common Name (CN) YR1
Organization (O) Let's Encrypt
Issued On Monday, August 10, 2026 at 10:01:51 AM
Expires On Sunday, November 8, 2026 at 9:01:50 AM
SingularCrane 20 minutes ago [-]
also seeing an expired cert:
Common Name
R12
Validity
Not Before
Tue, 10 Feb 2026 17:28:52 GMT
Not After
Mon, 11 May 2026 17:28:51 GMT
98codes 6 minutes ago [-]
This matches what I'm seeing.
ad_fontes 1 hours ago [-]
> Update - We've identified an issue with a database primary and are failing over to a replica immediately
Seems like a weird thing to post on a status page. Shouldn't this have happened automatically and therefore precluded the need to inform users of it?
thecosmicfrog 57 minutes ago [-]
> Update - primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues
everfrustrated 56 minutes ago [-]
>Update - primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues
And now they're blaming their upstream vendor! Embarrassing stuff to be writing on a public page.
AdrienPoupa 51 minutes ago [-]
I read that as an upstream service they own, but I agree the wording a bit weird.
heaney-555 51 minutes ago [-]
Why is that a problem if it _is_ an upstream vendor problem? (assuming it is)
theanonymousone 21 minutes ago [-]
The joke was a good one the first, second, or third time. It's not even funny anymore...
xray42 1 hours ago [-]
So a normal Wednesday
xbryanx 1 hours ago [-]
I spent a bunch of time during the outage last week setting up forgejo and some custom action runners. At the time, I was worried I was wasting time and getting distracted from my real work...alas, I guess not. Gonna finish up that work and complete the move today.
nickwanninger 21 minutes ago [-]
Same time next week?
Elfener 45 minutes ago [-]
Ah so that's why I got a random "github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks"
must feel bad that at this point every dev checks github status before going to work like it was the weather app.
esafak 12 minutes ago [-]
Cursor had better not miss this opportunity.
sevenseacat 41 minutes ago [-]
Was wondering why all my Actions just stopped running
qkwrv 1 hours ago [-]
We can't keep living like this.
pajamasam 59 minutes ago [-]
Apparently we can because a lot (most?) of us are still using GitHub even after all their outages recently.
nubinetwork 54 minutes ago [-]
Except nobody moves to a privately hosted "gitweb"...
kelvinjps10 50 minutes ago [-]
I have switched off from github to my own server besides two websites that depend on gitbub integration to deploy to clpudfare pages
acedTrex 1 hours ago [-]
Oh thank god my pink unicorn site is back online, its had great uptime lately so thats nice.
stalfosknight 44 minutes ago [-]
So what’s stopping you (or your org) from leaving GitHub?
iso1631 52 minutes ago [-]
Must be a weekday
55 minutes ago [-]
time0ut 1 hours ago [-]
Notice odd behavior on GitHub. Get gaslit by a green status page. Notice more odd behavior on GitHub. Think it must be me this time. See unusual action queuing. Ah, an incident on the status page. Go for a walk and check HN on my phone. The AI SDLC.
serial_dev 1 hours ago [-]
The status page is the last one to get the update. Reddit, HN, X, company chat are all always reporting it sooner.
zackify 1 hours ago [-]
Can't even run self hosted github actions lol
2 hours ago [-]
broyojo 23 minutes ago [-]
[dead]
everfrustrated 1 hours ago [-]
> We've identified an issue with a database primary and are failing over to a replica immediately
This is why it's hard to take GitHub seriously. How can a single database cause an outage for everyone? This is amateur stuff. Have they no sharding or partitioning internally? Paying customers should not be impacted in the same way as free ones are.
ZiiS 18 minutes ago [-]
2.9B commits per month; 100M action runs per day; I think they probably have some sharding.
ferguess_k 1 hours ago [-]
I wonder what is this database, and why it is hard to fall-over automatically.
inigyou 1 hours ago [-]
RDBMS replication and failover is way more difficult and manual than anyone would like. You can't just set up two postgres, tell them they're clustered and have it basically work; at a minimum you have to design the client to somehow know which one is currently the master, or use some sort of proxy (which becomes its own SPOF).
RDBMS integrity basically requires that one master server is responsible for the whole data set and other servers may replicate from it. And it usually doesn't wait for a quorum of replicas, just for one, because the design is to recover from a hardware failure, not a network partition, although that could be fixed at the cost of increased latency.
ferguess_k 3 minutes ago [-]
Thanks! I didn't get the chance to manage RDBMs but that's good to know.
croemer 55 minutes ago [-]
Possibly vitess from the latest update:
> primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues
ferguess_k 3 minutes ago [-]
Thanks!
rkozik1989 1 hours ago [-]
Did you not read it? Just because there's a database primary doesn't mean there is 1 primary database. There's likely man redundancies and they have issue with how they're allocating traffic to them which is in turn causing an issue with how much traffic redundancies are receiving.
inigyou 1 hours ago [-]
Why shouldn't it? Most companies run on a single database server. If they can immediately fail over to a replica, that's doing it right.
Maybe you expect that part of GitHub to have a scale where a single database can't handle it, but evidently that isn't true.
We can criticise them for not splitting up free and paid customers but again, most companies don't do that.
Which surprised me because of the many hundreds of services I rely on regularly that also have status pages.
Probably just a coincidence that Github started to have issues after beginning their move to Azure at the end of last year.
I'm told that GitHub has asserted to us that moving to this model means we would not be exposed to github.com outages. It's not at feature parity with github.com though.
It’s not the same thing as GitHub Enterprise Cloud hosted on the shared global network on github.com.
https://docs.github.com/en/enterprise-cloud@latest/admin/dat...
The real money is going to go towards, "make this all more reliable".
Their last postmortem made clear that their challenges are operational. Scale puts pressure on operation, but it's not what blocks them from keeping up.
Doubling the operation doubles the operational challenges.
I wonder if it was down that much, if users would get credits the way we pay when we use the services - its kind of ridiculous for a critical service to be down that much and all we do is "ah okay, its just github". Like, as if that was normal to be down that much...
Common Name (CN) www.dayswithoutgithubincident.com
Organization (O) <Not Part Of Certificate>
Common Name (CN) YR1
Organization (O) Let's Encrypt
Issued On Monday, August 10, 2026 at 10:01:51 AM
Expires On Sunday, November 8, 2026 at 9:01:50 AM
Seems like a weird thing to post on a status page. Shouldn't this have happened automatically and therefore precluded the need to inform users of it?
And now they're blaming their upstream vendor! Embarrassing stuff to be writing on a public page.
This is why it's hard to take GitHub seriously. How can a single database cause an outage for everyone? This is amateur stuff. Have they no sharding or partitioning internally? Paying customers should not be impacted in the same way as free ones are.
RDBMS integrity basically requires that one master server is responsible for the whole data set and other servers may replicate from it. And it usually doesn't wait for a quorum of replicas, just for one, because the design is to recover from a hardware failure, not a network partition, although that could be fixed at the cost of increased latency.
> primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues
Maybe you expect that part of GitHub to have a scale where a single database can't handle it, but evidently that isn't true.
We can criticise them for not splitting up free and paid customers but again, most companies don't do that.