@Duion You are correct, Bart didn't write all of Drupal, but he did write all of the *unique* code on this site. His drupal profile lists his public commits on those projects, but it is nowhere near all the code he has put into this site. I can see at least 5 entirely new and unique modules in the OGA module list, in addition to an entire unfinished project to update and rationalise the entire codebase. That is a lot of custom code for a new dev to learn.
The database size absolutely does matter, since it effects every non-cached dynamic request, of which there are a lot on this site. If you hit a Drupal server with a large number of concerrent non-cached requests against a large number of tables, you'll know about it in terms of page load times. It also makes the site more unwieldy to work with, because the larger your database, the longer Drush commands take to execute, so that even simple tasks can take hours waiting for a script to finish. In enterprise scenarios like on Acquia Cloud Site Forge, we deal with it using a massive front end Akamai cache to try and reduce the number of unique requests as far as possible.
The caching problem gets worse the more unique pages you make it possible for visitors to access, and the more tables each of those pages is calling on. The classic Drupal database chunkers are large views with lots of exposed filters, like the OGA search page. Just do the math on how many pieces of art there are, and then how many possible combinations of filters there are on the art search page. We'd have to be caching something like 3 x 1034 possible URLs, before we even add a search term.
Every user also has multiple pages which are each composed of yet more dynamic views, and we allow end users to create and populate an arbitrary number of new nodes and files. It really is a lot of possible ways for end users to directly utilise CPU time. None of these have any cost or disinsentive for the user, but they do have a cost on the server resources. if people are just downloading art and chatting on the forum that is generally fine, but if they start to do any actually intensive stuff, it can get to be a lot of concurrent intensive CPU jobs, which slow down everything for everyone,
Don't get me wrong, the bandwidth is also an issue, but that is easier to solve with small amounts of money than a CPU bottleneck, because of process locking.
@unergizer the site is built on Drupal, which is a very scalable open source solution used by millions of people including some of the biggest organisations on earth. It is used by NBC, Fox, Tesla, Twitter, Pintrest, Ebay and the US government, so it isn't a scalability or reliability issue, the difference is that those organisations have enough money to throw at servers and devs. I am sure there are ways OGA could be optomised, but you're never going to get past the fact that scaling up costs money.
The OGA server load comes from allowing a large number of users to perform complex database queries across a large quantity of files which each have heaps of dynamic metadata, and letting everyone upload and download as much as they want whenever they want. No matter which platform you build your site in, if you allow that much freedom to users you're going to run into either performance issues or high server costs.
Most other file sharing sites have per user download limits, or tiered paid accounts, or they sell user data, or just have lots of adds. What other site lets you upload and download such a huge variety of file types so freely? This site is like a cross between a social network, megaupload, deviantart and soundcloud. What do you think their hosting and support costs are like? How do they pay for it?
Home page - Latest art expanded it to 8 items as requested - may as well see how it goes, though I can already see a cross over between latest and popular, which is quite likely on slow weeks.
Home page - title link is split out into "view more" to address concerns that some visitors aren't noticing the links. also makes the titles easier to read. - I am a little concerned it adds clutter to the page.
Latest art page - "view more" link now correctly takes you to the latest art of that type.
Latest art page - "view more" link moved from bottom right to next to title to match the home page - also decreases screen space used by each element
Main menu "browse" links now take you to the most popular art of that type.
Yeah I should clarify, I don't actually know the exact hosting costs and I don't feel comfortable speaking on Bart's behalf about money. I absolutely appreciate how amazing it is that the site is getting donations, and I am personally very grateful to each and every person who helps out with their hard earned dollarydoos.
That said, it is worth acknowledging that as amazing as it is, the current monthly funding rate is less than the daily rate of pay for a Drupal developer in Australia, and less than my rent for a week, I imagine its similar where Bart lives. It may be worth investigating hiring a developer from elsewhere in the world, but given Bart wrote the code for the entire site, he does seem like a logic choice to work on it.
In terms of server load, there are something like 27,000 approved users with profiles and collections and favourites, which is similar to a medium sized forum. The difference is that there are something like 30,000 art entires, each of which can have dozens of files which can each be hudreds of megabytes. The closest analogy would be a media sharing site like a small version of deviantart, soundcloud or youtube. The server load for those kinds of sites typically requires a lot more ads, corporate backing or paid premium accounts. Drupal scales well, but any database can get a bit unwieldy when you regularly do complex queries against multiple tables with millions of cells.
As for open sourcing the code, I know that is the long term intention and one of the patreon funding goals, There is work involved, there are custom modules which need a bit of documentation, unit tests and clean up, and I think the new theme was going to be finished first. There will also be some overhead in setting up and managing the dev community along with testing and accepting new code. It is not a huge amount of work, but it is probably at least a weeks worth. Even then, that isn't going to solve everything, Drupal doesn't handle syncing database commits very gracefully, there is a lot of webserver and app config work that will still have to be manual.
If you're interested in the funding goals and motivations, I recomend reading through Bart's topic
capbros. Essentially the situation is this. Based on the Patreon funding drive, the community haven't deemed it worthwhile to fund OGA development, so there has been no OGA development.
I have a young family, a more than full time job and many other interests (like a sweet new VR headset). I can't speak for other admins, but i believe Bart is in a similar situation and all of our admins and mods are volunteers with other commitments.
I am happy to do jobs around the place which take a few of hours on the odd weekend I get free, but I can't really dedicate a solid week to something like setting up and configuring a new Solr search instance.
If we as a community can raise enough money to actually pay Bart's bills, he has graciously said he would end his current contract and work on the site full time. That has not happened, so development has stagnated. As I understand it the current rate of funding and advertisements doesn't actually cover the hosting costs, so Bart is essentially paying for this site to stay up out of the goodness of his heart.
So we are at an impasse, either we find a new trusted volunteer admin with heaps of free time who has extensive experience doing PHP development and is familiar with Drupal 7 and Apache Solr, or we stump up the funds to pay for it. Alternatively we take the path of least resistance and keep doing what we're doing, break fixes and minor improvements.
I started work on a redesign where i changed the padding, and I believe Bart already did a heap of work on a redesign as well. I am guessing Bart was hoping to work on the redesign and release it if the patreon funding ever became high enough to allow him to spend more time on it. As it happens, the patreon funding has stopped at about a tenth of what it would cost to have a full time dev, which means we haven't got the planned redesign either.
It rather depends on the day and time, sometimes the new queue seems to move at a glacial pace. If we set it to 8 items it looks like nothing ever changes at those times.
Hi Melaniko, the main thing that struck me was the lack of sound effects. Some ambient environmental sounds would be a start, but sounds in reaction to events would be a big addition. In particular, air rushing in reaction to changes in your acceleration, sounds of the bats dying and the vehicles going past. It would generally increase the impact of player actions.
@Duion You are correct, Bart didn't write all of Drupal, but he did write all of the *unique* code on this site. His drupal profile lists his public commits on those projects, but it is nowhere near all the code he has put into this site. I can see at least 5 entirely new and unique modules in the OGA module list, in addition to an entire unfinished project to update and rationalise the entire codebase. That is a lot of custom code for a new dev to learn.
The database size absolutely does matter, since it effects every non-cached dynamic request, of which there are a lot on this site. If you hit a Drupal server with a large number of concerrent non-cached requests against a large number of tables, you'll know about it in terms of page load times. It also makes the site more unwieldy to work with, because the larger your database, the longer Drush commands take to execute, so that even simple tasks can take hours waiting for a script to finish. In enterprise scenarios like on Acquia Cloud Site Forge, we deal with it using a massive front end Akamai cache to try and reduce the number of unique requests as far as possible.
The caching problem gets worse the more unique pages you make it possible for visitors to access, and the more tables each of those pages is calling on. The classic Drupal database chunkers are large views with lots of exposed filters, like the OGA search page. Just do the math on how many pieces of art there are, and then how many possible combinations of filters there are on the art search page. We'd have to be caching something like 3 x 1034 possible URLs, before we even add a search term.
Every user also has multiple pages which are each composed of yet more dynamic views, and we allow end users to create and populate an arbitrary number of new nodes and files. It really is a lot of possible ways for end users to directly utilise CPU time. None of these have any cost or disinsentive for the user, but they do have a cost on the server resources. if people are just downloading art and chatting on the forum that is generally fine, but if they start to do any actually intensive stuff, it can get to be a lot of concurrent intensive CPU jobs, which slow down everything for everyone,
Don't get me wrong, the bandwidth is also an issue, but that is easier to solve with small amounts of money than a CPU bottleneck, because of process locking.
@unergizer the site is built on Drupal, which is a very scalable open source solution used by millions of people including some of the biggest organisations on earth. It is used by NBC, Fox, Tesla, Twitter, Pintrest, Ebay and the US government, so it isn't a scalability or reliability issue, the difference is that those organisations have enough money to throw at servers and devs. I am sure there are ways OGA could be optomised, but you're never going to get past the fact that scaling up costs money.
The OGA server load comes from allowing a large number of users to perform complex database queries across a large quantity of files which each have heaps of dynamic metadata, and letting everyone upload and download as much as they want whenever they want. No matter which platform you build your site in, if you allow that much freedom to users you're going to run into either performance issues or high server costs.
Most other file sharing sites have per user download limits, or tiered paid accounts, or they sell user data, or just have lots of adds. What other site lets you upload and download such a huge variety of file types so freely? This site is like a cross between a social network, megaupload, deviantart and soundcloud. What do you think their hosting and support costs are like? How do they pay for it?
I have made the following changes:
Yeah I should clarify, I don't actually know the exact hosting costs and I don't feel comfortable speaking on Bart's behalf about money. I absolutely appreciate how amazing it is that the site is getting donations, and I am personally very grateful to each and every person who helps out with their hard earned dollarydoos.
That said, it is worth acknowledging that as amazing as it is, the current monthly funding rate is less than the daily rate of pay for a Drupal developer in Australia, and less than my rent for a week, I imagine its similar where Bart lives. It may be worth investigating hiring a developer from elsewhere in the world, but given Bart wrote the code for the entire site, he does seem like a logic choice to work on it.
In terms of server load, there are something like 27,000 approved users with profiles and collections and favourites, which is similar to a medium sized forum. The difference is that there are something like 30,000 art entires, each of which can have dozens of files which can each be hudreds of megabytes. The closest analogy would be a media sharing site like a small version of deviantart, soundcloud or youtube. The server load for those kinds of sites typically requires a lot more ads, corporate backing or paid premium accounts. Drupal scales well, but any database can get a bit unwieldy when you regularly do complex queries against multiple tables with millions of cells.
As for open sourcing the code, I know that is the long term intention and one of the patreon funding goals, There is work involved, there are custom modules which need a bit of documentation, unit tests and clean up, and I think the new theme was going to be finished first. There will also be some overhead in setting up and managing the dev community along with testing and accepting new code. It is not a huge amount of work, but it is probably at least a weeks worth. Even then, that isn't going to solve everything, Drupal doesn't handle syncing database commits very gracefully, there is a lot of webserver and app config work that will still have to be manual.
If you're interested in the funding goals and motivations, I recomend reading through Bart's topic
http://opengameart.org/content/support-opengameartorg-on-patreon
capbros. Essentially the situation is this. Based on the Patreon funding drive, the community haven't deemed it worthwhile to fund OGA development, so there has been no OGA development.
I have a young family, a more than full time job and many other interests (like a sweet new VR headset). I can't speak for other admins, but i believe Bart is in a similar situation and all of our admins and mods are volunteers with other commitments.
I am happy to do jobs around the place which take a few of hours on the odd weekend I get free, but I can't really dedicate a solid week to something like setting up and configuring a new Solr search instance.
If we as a community can raise enough money to actually pay Bart's bills, he has graciously said he would end his current contract and work on the site full time. That has not happened, so development has stagnated. As I understand it the current rate of funding and advertisements doesn't actually cover the hosting costs, so Bart is essentially paying for this site to stay up out of the goodness of his heart.
So we are at an impasse, either we find a new trusted volunteer admin with heaps of free time who has extensive experience doing PHP development and is familiar with Drupal 7 and Apache Solr, or we stump up the funds to pay for it. Alternatively we take the path of least resistance and keep doing what we're doing, break fixes and minor improvements.
I started work on a redesign where i changed the padding, and I believe Bart already did a heap of work on a redesign as well. I am guessing Bart was hoping to work on the redesign and release it if the patreon funding ever became high enough to allow him to spend more time on it. As it happens, the patreon funding has stopped at about a tenth of what it would cost to have a full time dev, which means we haven't got the planned redesign either.
It rather depends on the day and time, sometimes the new queue seems to move at a glacial pace. If we set it to 8 items it looks like nothing ever changes at those times.
Damn, that is pretty cool. Nice work Reemax!
Do you have a todo list, dev roadmap or art wishlist?
Hi Melaniko, the main thing that struck me was the lack of sound effects. Some ambient environmental sounds would be a start, but sounds in reaction to events would be a big addition. In particular, air rushing in reaction to changes in your acceleration, sounds of the bats dying and the vehicles going past. It would generally increase the impact of player actions.
keep it up!
I have added a short note to the file upload field which simply says
"Editable source files are welcome and will make your work usable in more projects"
Pages