Wikipedia talk:Avoid using meta-templates/Archive 1

Page contents not supported in other languages.
From Wikipedia, the free encyclopedia

Portions of this page come from comments made by User:Jamesday and myself on Template talk:Sisterproject. -- Netoholic @ 19:05, 2005 Feb 4 (UTC)

What's the real problem?

Doesn't most of this apply to any popular template? In other words, if Wikipedia:WikiProject Stub sorting had not begun, or were to be called off, wouldn't this apply to the original {{stub}} template as well? — Itai (f&t) 14:41, 7 Feb 2005 (UTC)

Certainly, the potential for vandalism exists, and the template could be protected for those reasons. The server load problem still partially exists because of the use of the category in that template. Meta-templates are worse though. -- Netoholic @ 16:00, 2005 Feb 7 (UTC)
Could be. Basically, what I'm saying is this: are we sure this is not part of a greater problem? Seems that the problem is not meta-templates per se, but rather any template that is used on more than n pages. — Itai (f&t) 16:55, 7 Feb 2005 (UTC)


Unfair discrimination against registered users?

It seems to me that a large part of the problem stems from the different treatment given to anonymous users, who are served by massive caches which must be invalidated every time an edit is made. Registered users do not reap the benefit of these caches, because of the way the software implements the "logged-in-ness" of a given user.

Maybe some effort should be applied to making the software more efficient for registered users, if necessary at the expense of worse performance for anonymous users, which would encourage more people to log in.

I'm not expert enough in the whys and wherefores of XHTML/CSS to understand quite why anonymous users should be able to use caching—and logged-in users cannot—when I thought the real difference is in the CSS "wrapper" which should be independent of the "real content".

Enlighten me, someone, please!

--Phil | Talk 11:35, Feb 8, 2005 (UTC)

Happy to.:) When a page has been removed from caches, the first view of the page causes several steps significant to this discussion:

  • Each item in the base page portion is requested from the database (images and CSS aren't in the main part). The page you edit, each templage, each template included in the template and so on. Two templates, two database records to be retrived. One template on its own, one read, one template including another, two. Plus the one for the base page.
  • Once that and the rest of what is called the parsing is done, the page is saved in the parser cache. That's kept in RAM in memcached, subject to a capacity limit, currently 12 memcached programs running, ach holding 280MB for a total of 3.3GB of cache (a bit is used for other things but this is most of it). I see that 8 more are currently disabled - probably a temporary problem. In a few weeks I expect it to be about 12-15GB for this.
  • Finally the skin is applied and the page is passed on to the Squids, which cache it for all who aren't logged in (will only be useful if it's the normal skin) and send it on to the person who originally requested it.
  • Whenever any part of a page is changed, be it the page itself or a template or image used in it, the page is marked as changed and will be regenerated next time it is requested. Both the Squid and parser caches have it removed. Necessary so people see the correct version.

The Squids, because of the limitations in the way they can work, with much less work per page, are inherently the fastest way of serving the pages and just 4 machines can serve some 75% of all hits to the site. But they are restricted in what they can serve. Next step is using the parser cache via the apache web servers. That allows all of the user settings for logged in people but uses more web server CPU so it's less efficient. But can't yet be avoided (we're working on it - we want to serve as much as possible from cache).

We could switch everyone to using the apaches but that would be far less efficient and would require something like 4-5 times as many apaches as we have today, far more than the 4 machines gained by not using them as squids.

So, logged in people aren't being discriminated against. They also get caching. But the need to support user preferences does mean that they can't be served as efficiently as those who aren't logged in. Jamesday 08:09, 13 Feb 2005 (UTC)

I don't understand why you need to reparse the page in order to support user preferences. There are only two preferences that affect layout (3 options for dates and 6 options for math). So why not include all 9 options in the document, using CSS and JavaScript to select which one gets presented in the browser? Then the same cached page would work for all users, logged in or not. Gdr 16:07, 2005 Mar 22 (UTC)
Work is ongoing to try to do more with CSS. Other custom settings are different skins and stub display threshold (though that may be disabled at present and might not return). There are more in the misc settings area of preferences, though script could do some. We'retrying to cache everything we can cache, because that is so much more efficient. Jamesday 15:24, 7 Apr 2005 (UTC)
What does
Two templates, to database records to be retrived.
One template on its own, one read, one template including another, two.
Plus the one for the base page.
mean? MarSch 16:36, 26 Mar 2005 (UTC)
I've corrected the typo of the "to templates". Should have been "two templates". That is, each template is one database read. Jamesday 15:24, 7 Apr 2005 (UTC)

This policy is still being considered

Just in case anybody was not clear on the subject - this policy is, at the moment, being considered. It has not been endorsed by the Wikipedia community. No decisions should be based on it. — Itai (f&t) 22:29, 8 Feb 2005 (UTC)

This is not intended to be "policy". The Be bold guideline isn't "policy" either, but that doesn't mean we should ignore its wise guidance. -- Netoholic @ 17:25, 2005 Feb 9 (UTC)
The wiseness of this non-policy is being debated as well. In other words: don't point people, through edits summaries or directly, to this page as an explanation for actions without pointing out that this is not an official policy, merely the opinion of some Wikipedians. — Itai (f&t) 17:38, 9 Feb 2005 (UTC)
I see no debate refuting the points made in this page. Your argument on other talk pages seems to be about templates in general, but that doesn't mean the assertions of this page are incorrect. Beyond simple vandalism, meta-templates have other technical problems, which are documented here. -- Netoholic @ 18:22, 2005 Feb 9 (UTC)
Not policy but Netoholic is right in suggesting that templates in general and more so, templates within templates, increase server load . Better not to use meta templates if relatively few templates are involved and there's much concern for efficiency and page loading times. Jamesday 08:09, 13 Feb 2005 (UTC)
Can you explain a little more on why templates in general are a problem, and how much of a problem they are? BlankVerse ∅ 13:08, 13 Feb 2005 (UTC)
Did that on the page and also added some more links to it. Jamesday 15:24, 7 Apr 2005 (UTC)

Possible page split

As I mentioned above, I'm not sure most of the content of this page is directly related to meta-templates. How about moving everything that is not directly related to, for instance, Wikipedia:Problems with frequently used templates (along with a note saying that meta-templates often fall under this criterion), and keep on this page (which I would rather have moved to Wikipedia:Problems with meta-templates, but this really isn't important) only what relates to meta-templates (along with a note pointing out that meta-templates are often frequently used). — Itai (f&t) 19:18, 10 Feb 2005 (UTC)

I've tried to assist by highlighting some of the general "big template" issues and the ones whcih are specific to templates wihtin templates. Basically, templates within templates because they are likely to affect more pages, do have a larger effect than is initially apparent, mostly because of that lag issue, but also because of the extra base load for the database servers. Base load we can buy more machines for, that lag is a far tougher problem. You cna probably extract the generic template bits for a page on that. Jamesday 08:09, 13 Feb 2005 (UTC)
Whatever is true of regular templates is twice as true for meta-templates. Please don't split out sections of this page, and don't rename it. -- Netoholic @ 20:55, 2005 Feb 10 (UTC)
I've had no intention of taking any such actions unilaterally. However, I still maintain that this page is misleadingly named. This is like a Ford advertisement saying: "you shouldn't buy Hondas because Hondas can't fly." True, but the same can be said of all other automobiles. Wouldn't it be better to have a site-wide policy regarding all frequently used templates (say, those having over 10,000 occurrences), and keep this page for problems restricted to meta-templates? (Along with a note saying that some meta-templates also have the misfortune of being frequently used, and thus pose the problems mentioned on the newly created page.) — Itai (f&t) 21:58, 10 Feb 2005 (UTC)
If you like, I'm sure that when template parameters were first considered, there was a lot of discussion regarding what's wrong with parameters. You can add the arguments against allowing parameters to this page. (Meta-templates generally use parameters.) In fact, if you wish to make this page longer still, you can even include bits about what's wrong with wikis in general - which applies, admittedly, to meta-templates as well. — Itai (f&t) 21:20, 11 Feb 2005 (UTC)
I've no significant load concerns about parameters in templates. The parsing cost is minimal compared to the extra database reads of templates or templates within templates. People may have stylistic and taste objections but that's a different thing. Jamesday 08:09, 13 Feb 2005 (UTC)
Itai, stop trolling. This page presents many of the major issues of meta-templates. Vandalism is only one, but it's impact is felt even harder than when using one template. For example, if I have meta-template that governs 20 other templates used on 100 pages each (2000 pages affected), it is more damaging to vandalise the one meta-template than it would be to vandalize a single template used on those 2000 pages. -- Netoholic @ 21:48, 2005 Feb 11 (UTC)
Netoholic, stop murdering infants. (You're not, but neither am I trolling.) If using a meta-template-generated templates for these 2000 pages is indeed worse than using a single template, list the reasons for this on this page. You've still to provide one good reason why it wouldn't make more sense to list the problems with all frequently used templates on one page, and keep this page, which has "meta-templates" in its title, for problems specific to meta-templates. (Including, for instance, the reasons for your above statement.) — Itai (f&t) 02:08, 12 Feb 2005 (UTC)
Each meta-template doubles the number of template records which must be retrieved from the database when the page is being built, slowing down the page view and adding database server load. That's true regardless of caching questions. If a meta-template is used in 5,000 other templates it may be worth having anyway because of the amount of work it saves (though it's probably debatable whether it is really wanted by all 5,000). If it's used in 10, twice the number of template retrievals seems like a high cost to pay compared to the human work saved. We can solicit donations to buy more hardware to do that work, but it's nice to try to be efficient as well. For this reason, meta-templates are a special, more costly, case of the general big template situation and do merit some extra consideration. Jamesday 08:09, 13 Feb 2005 (UTC)
What if we quit using those templates that are currently being used as meta-templates as meta-templates, but just use them only as a common template design, and then create the "daughter" templates using {{subst:meta-template}}? That should get rid of the double read.
From your description, it also sounds like we should be trying to keep down the number of templates per page. I've seen a few pages that have been plastered with an attention template, a dispute template, and a stub template. Should there be a policy that in general there should only be one (or two?) templates/page?
Although I haven't seen you address the issue, I am guessing that there may also a problem with some of the very large templates that are sometimes being created (compared to small templates)? Also: How easy would it be to get a list of the most commonly used templates with an estimate of the number of pages they are being used on? BlankVerse ∅ 09:47, 13 Feb 2005 (UTC)
Your analysis is correct in all respects. However, if a page needs multiple templates, then it needs them - it's just something we want to try to do efficiently. I wouldn't want a fixed rule saying no more than two templates per page. templates are an excellent tool. People are just being a bit too enthusiastic with them at the moment.:) Don't actually have to use a template every time something is common - only when it's likely to be necessary to change it fairly regularly. Jamesday 15:24, 7 Apr 2005 (UTC)

Use lists, not templates and categories?

As I discussed at Wikipedia:Templates for deletion#Template:us-geo-stub, using lists, because of the large number of entries, and the large amount of human maintenance that would be required, is an unworkable alternative for trying to deal with stub articles. The topic stubs were created because the Category:stub had grown so huge that it had to be disabled because of the hit on system performance. The Category:substubs was also growing to an unmanageable size. Regrouping the stubs into smaller stub categories allows individuals who are interesting in those topics, plus the Regional notice boards and Wikiprojects, to find and expand those stubs into larger articles. As far as I can see, the best solution for handling the topic stubs may be to quit use the metastubs except that the text for the metastubs is saved somewhere so that they can be used as text templates for creating topic stubs with a standardized format. BlankVerse 09:37, 11 Feb 2005 (UTC)

Can I ask this... why are unsorted stubs such a problem? Take us-geo-stub, for example - I can guarantee there is noone sitting around saying "Gee, I think I'll see what random U.S. locations are stubs, and improve one". Let's leave them to be happy little stubs, sitting out there doing no harm and not being referenced, until someone actually interested in that subject comes along and improves it. Sorting stubs is busywork. For any specific topic area, wouldn't it make more sense to just make a list of the needed articles, and prioritize them for work? I'm not saying you must maintain a list of every related stub, but why go out searching for them just to categorize them? The problem is, the stub message should never have had the Category:Stub added to it. We only needed a simple text to encourage people to come in and edit. -- Netoholic @ 15:43, 2005 Feb 11 (UTC)
1) I have already explained to you in the Wikipedia:Templates for deletion#Template:us-geo-stub discussion that I have used the Template:Japan-stub/Category:Japan-related stubs combination to look for articles on Japanese topics to un-stub. I've heard of other Wikipedians doing similiar things for other subject areas. So yes, there are people who are using the topic stubs to help un-stub articles.
2) Leaving the articles as just stubs will guarantee that many of them will NEVER be upgraded unless someone comes along interested in that specific article rather than that subject area.
3) Having a huge number of articles with template:stub has been a problem because category:stub because so huge that it was big performance hit and so the stub category had to be removed from that template. That means that the only way to find those Wikipedia articles with the stub template but are not in the stub category is to do a Google site search.
4) Lists would be a HUGE amount of busy work if there were to be used to keeping track of stubs. BlankVerse ∅ 13:04, 13 Feb 2005 (UTC)
I can guarantee there is noone sitting around saying "Gee, I think I'll see what random U.S. locations are stubs, and improve one". Can you now? I don't spend a whole lot of time looking at the geo-stubs, but I do check in occasionally to see if there are any that I recognize and can help out. However, while I find the geo-stubs marginally useful, I'm really not convinced about the entire elaborate new stub sorting apparatus. Seems unnecessarily complex to me, but then there may be others who, like me, check for stubs in their areas of interest from time to time. So count me as decidedly ambivalent. I can see how a *simple* stub sorting method is superior to making lists (it is trivially easy for a random editor to add a stub tag to an article, while adding the article to an appropriate list is considerably more involved). older≠wiser 16:03, Feb 11, 2005 (UTC)
I agree with you. That guarantee was pretty much a blatant assumption; all editors don't (and shouldn't) edit the same way. That is exactly what people are doing. I often pick a stub topic and go through it, copyedit, google it, get out a book I have on the subject. It's pretty difficult to find articles that really need development otherwise. Randomly clicking hoping to find an article in the subject I'm interested that really needs it is a waste of my limited time. --Sketchee 16:05, Feb 28, 2005 (UTC)
Personally, not speaking as the developer most responsible for database issues for this comment, I'd like to see stubs in both alphabetic and subject categories (with alphabetic grouped by the first 3 letters of the page name). There are people who like to work on specific topic areas and also are people who like to just work on random stubs. (shrug) Easy enough to help both groups. There's no extra cost for this in the individual page views. Alphabetic grouping for the full stub list to keep the category sizes manageable. Jamesday 08:09, 13 Feb 2005 (UTC)
Now as a developer: large categories were a significant load problem. They are still an issue but less so and may be a smaller factor with MediaWiki 1.5, whenever that arrives. For load reasons it's nice to keep the total number of items in a category in the few thousand range. For general guidance, we use 500 for the all pages lists and such because 500 is currently about the maximum normal comfort level. More than that starts to become an issue and in the thousands I start to notice them showing up as unusually slow queries which are delaying others. Try to think how big the category will eventually be and segment it in some way with a target in the 500 or fewer members range if you're trying to avoid load issues. In November or December 2004 I requested an emergency change in how categories were handled (hard limit to 500 members). I also acted as a developer to temporarily protect one template at a version without a category or image in it for load reasons. Another change made it possible to remove that limit and protection, though it's an imperfect solution at present - just good enough to dodge a hard limit. There's a change coming in MediaWiki 1.5 which may raise the 500 number by reducing the cost a bit. Jamesday 08:09, 13 Feb 2005 (UTC)
Can you tell us how many Wikipedia articles currently have the stub and substub templates? Also: which categories on the Wikipedia currently have the most number of articles? Finally: Are images in templates a performance issue? BlankVerse ∅ 13:04, 13 Feb 2005 (UTC)
Can't count right now. Too many for comfort. Images in templates are a significant performance issue at present. Work to change the software to try to make them less of an issue is happening. Have been two tries so far because the image server load was making the site comletely unavailable at times. Temporarily removing the images from the most popular templates helped quite a bit. See [1]. Too soon to say whether the second software approach will help enough - buying another, more powerful, server anyway, I think, because it's a critical problem we can't allow to continue. It might end up leading to us turning off image display completely just to keep the sites available if we don't get it dealt with. Temporarily removing the images from the most popular stubs did enough good that we haven't yet been forced to do that. Hopefully either the software changes will do enough good or the new hardare will do enough good before we have to do something more dramatic. Longer term more software changes are coming (distributing the images over many servers). Jamesday 15:24, 7 Apr 2005 (UTC)

Scope creep

Netoholic: Your description was overblown as well as incorrect. I still left your basic points intact. Without the exaggerations that were in your original version, people will actually probably believe your position more. BlankVerse ∅ 17:36, 12 Feb 2005 (UTC)

There are no exaggerations. The wording may be "dramatic", but this page serves to describe a strong position against the use of these meta-templates. -- Netoholic @ 01:00, 2005 Feb 13 (UTC)

From your description in the Scope creep section:

  • Baring mind, if we have 250 stub templates for roughly 475k articles, how many do you think we'll have when the Wikipedia reaches 1 million articles? Certainly not 250 stub templates. Maybe 500, maybe more. We might have as many stub templates and stub categories as we do have categories now. -- AllyUnion (talk) 06:31, 16 Feb 2005 (UTC)
    You do relise you just made a powerful argument for being worried about the issue, by pointing out that the number of pages affected by each meta templates is going to continue growing? Scaling issues are a major concern. Not the immediate one, but yes, your point is valid and suggests that we need to be trying to work effiiciently. Not what you intended it to mean, though. What breaks if the 500 stub templates don't have the latest possible version of one or more of the meta templates because subst has been used? Do the pages cease being stubs? How did we survive without being able to use templates? The answer to that is: they really don't need to be current, because they still do their job if they aren't. Jamesday 15:24, 7 Apr 2005 (UTC)
  • "many of which have been created on the spur of the moment": False. If you had check the various pages for the Wikipedia:WikiProject Stub sorting, you would have seen that the new topic stubs were being created with specific needs in mind. For example, many of them are being created to be associated with different WikiProjects or Regional notice boards.
  • It was spur of the moment when it started, by the way. Certainly not completely the case now, as project members are trying to keep in mind not to create any stub without asking first. -- AllyUnion (talk) 06:31, 16 Feb 2005 (UTC)
  • "... for use on only a few articles": False: The stubs that are being created by the people in the Stub-sorting WikiProject are only being created when it can be shown that there are enough stubs to populate the stub category. Although things were a little too haphazard to begin with, especially before the creation of the Stub-sorting WikiProject, there is now a Wikipedia:Stub sorting policy, which says there should be a minimum of 100 articles.
    Baring in mind, that was Netoholic's idea. -- AllyUnion (talk) 06:24, 16 Feb 2005 (UTC)
    It use to be 10, by the way. -- AllyUnion (talk) 06:24, 16 Feb 2005 (UTC)
    And oh, that stub sorting policy has not been made official, and still in the draft stages. -- AllyUnion (talk) 06:31, 16 Feb 2005 (UTC)
  • "many of these new "child" templates have come up for deletion": False. There have been a few "joke" templates at WP:TFD (such as Template:Cowstub and Template:Stub-sorting-stub), there have been a few accidental template creations that duplicated the scope of existing stub templates (such as Template:us-geo-stub and Template: Car-stub), and there have been a few that were created where the Stub-sorting WikiProject later decided that there were better ways to categorize things (such as Template: City-stub). Finally, there was also Template:Bush-stub, which was created by someone not involved with the Stub-sorting WikiProject. You've made it sound like there were dozens of stub templates that were inundating the WP:TFD process.

In my opinion, your descriptions went beyond just exaggerations in some cases. They were inaccurate descriptions of the stub template creation process, which it appears you have not investigated before you created your descriptions. BlankVerse ∅ 12:27, 13 Feb 2005 (UTC)

AllyUnion's responses are spot-on. Leave this section alone. Feel free to write Wikipedia:Meta-templates are teh best or somesuch. This page is to describe the negatives of using them, and no watering down of the scope creep section is welcome. -- Netoholic @ 04:26, 2005 Feb 18 (UTC)
I believe that meta-templates are considered harmful if and only if they are not used with the subst: included. Netoholic's example of the stub sorting is valid simply because it's one of the best examples he could find. I doubt that Netoholic would object if every stub template used {{subst:metastub}} instead of {{metastub}}. -- AllyUnion (talk) 22:17, 19 Feb 2005 (UTC)
If -all- that a template is used for is to "subst:", then I don't think the template should exist. In general it is much better to document the wikicode for the template "genre" on a project page for easy copy and pasting. -- Netoholic @ 03:29, 2005 Feb 20 (UTC)
Then we should delete {{vt}}, {{vb}}, {{vfd top}}, {{vfd bottom}}, {{vfd}} under your logic. -- AllyUnion (talk) 07:07, 4 Mar 2005 (UTC)

Template:Message box

Stub templates aren't the only meta-templates in use. See Template:Message box. -- AllyUnion (talk) 07:08, 4 Mar 2005 (UTC)

Question about caching.

As the stub messages are being turned from readable semantically-marked-up code to less-readable stuff with DIVs flyign to and fro, I gotta as this.

As Jamesday pointed out, when a page's entry in the cache is empty, a database query is made to fetch that page. Then another one for each template it contains, and so forth. Why on earth aren't we caching the templates as well?

From what Jamesday said, this is the current process: Say page1 contains template, which contains meta-template. page2 also contains template. Then, if meta-template is changed, loading the two pages causes this to happen:

  • page1(template(meta-template)) is dirty, needs to be refreshed. Fetch page1 from db.
    • Fetch template from db.
      • Fetch meta-template from db.
    • Assemble template(meta-template).
  • Assemble page1(template(meta-template)).
  • Serve (and cache) page1(template(meta-template)).
  • Fetch page2 from db.
    • Fetch template from db.
      • Fetch meta-template from db.
    • Assemble template(meta-template).
  • Assemble page2(template(meta-template)).
  • Serve (and cache) page2(template(meta-template)).

This can't possibly be right, but it looks like this is what Jamesday is saying is happening. If so, no wonder meta-templates are such a performance hit. What's wrong with caching a version of the page with references to the templates, so they can be substituted into an already in-cache page without dirtying it? So we'd avoid dirtying zillions of pages and requiring them to be fetched all the way from the database again when, really, nothing's actually changed in them.

Am I way off base here? Someone set me straight. grendel|khan 02:07, 2005 Mar 4 (UTC)

  1. It seems like we can change the way pages are cached instead of inconveniencing everyone by removing meta-templates. True?
  2. If "glorified stylesheets" are converted into css classes, the classes have to be as visible and easy to use as the templates and also editable. - Omegatron 23:41, Mar 21, 2005 (UTC)
Eventually that can be done. Doesn't change what's happening at present, though. One issue we were discussing a few days ago was the cost of lots of round trips to the cache server for each page build and the desirability of avoiding those, particularly as we move to building sites at different data centers. That forced us for now to want to stick to sites very close to the current Florida location, because of the delays involved with lots of sequential cache requests at perhaps 200ms each from korea. Asking the database slave which could be located in Korea would actually have been better than using the different memcached cache for that. Not ideal and I hope that will also change.
Where would you do the cache lookups? Options are:
  • Squids (which would increase the cost per page for the squids and is something we can't currently do anyway because what we have doesn't support it
  • Apaches and memcached. Could do. 30 or so of them are needed to do the 25% of work the Squids don't do. Is four times as many an improvement? Probably not, so wouldn't want to do it all the time, which leaves the squids caches being purged and them reloading from an apache as at present. I don't know whether the parser cache currently saves expanded or non-expanded templates - not expanding them and caching them individually might be an improvement. Certainly could cache the templates if they aren't currently cached and that would help with the page views for those who are logged in and for the rebuilds and database queries. Jamesday 15:24, 7 Apr 2005 (UTC)


move to guideline status

No one's made any major changes to this in a while, nor refuted the basic conclusions. We already have some areas (like stubs) which are moving away from meta-template schemes now that the technical reasons have been described. One of the main reasons cited for other areas which are not yet adopting this principle is that it is not "policy". Well, let's start moving that way.

I'd like to move this from an "opinion" piece to a guideline. Namely, I propose renaming this to something like "Avoid using meta-templates", and replace the "proposed" line with one specifying this as an active guideline. I'd like to also link to it from the relevant template-related pages. -- Netoholic @ 22:31, 2005 Mar 21 (UTC)

The discussion has become moribund here not because there has been any consensus reached, but instead, it is because you have taken ownership of the article and not allowed any changes that you do not agree with. For example, your censorship of any mention of the use of "subst:". I do agree that Meta-templates should not be directly used, but without the use of such templates as the metastub template, you had the problem that existed before it was created where the newly created topic stubs had different wordings and formattings. Since the participants in the Stub-sorting Wikiproject have quit using the metastub template, that problem has returned.
I think that this is close to being ready to be Wikipedia:Semi-policy, but I think that it still needs some work and some rewording. Without those changes, I do not support changing its status. It certainly isn't ready to be official Wikipedia policy yet, which is what it sounds like you want to do. (Just a suggestion: I think that you need to become more familiar with Wikipedia:How to create policy, including "Do not rush" and "Consult widely - make a special effort to engage potential critics of the new policy".) BlankVerse ∅ 13:02, 22 Mar 2005 (UTC)
I am well aware of how to create policy, and don't see this as rushing. Please propose what you'd like to add. -- Netoholic @ 14:59, 2005 Mar 22 (UTC)
Except for mention of ":subst", it is mostly deletions that I think need to be done to the proposal. For example, since User:Jamesday has said that creating topic stubs is a good thing™, then the entire "Scope creep" section (or at least any mention of the topic stubs) weakens your arguments. "Use lists" under "Alternatives" should be removed because it is an unworkable proposal. There also needs to be some discussion of using ":subst" as a way to use a template as a standard format (e.g. Metastub), but still avoid the double database call. I'll think about it some more, and then do some editing in a day or two.
I do agree that the article name should be changed to either "Avoid using meta-templates" or just "Avoid meta-templates". BlankVerse ∅ 03:03, 23 Mar 2005 (UTC)
I don't even understand why this is necessary. Have you asked for opinions in places where people are likely to see it? - Omegatron 16:17, Mar 22, 2005 (UTC)
Well, because I think its important that we have something to back up the contention that "meta-templates" should in fact not be used. In its current state, people consider more of an opinion and optional. Moving to guideline makes for a stronger stance. -- Netoholic @ 16:53, 2005 Mar 22 (UTC)
Turning it into a guideline doesn't justify it. I'm asking if you've convinced a large number of people that meta-templates are inherently bad and can't be fixed in software. I'm not convinced that this is necessary in the first place. I don't see any kind of consensus from this talk page. - Omegatron 17:41, Mar 22, 2005 (UTC)
Things are suffering today. It's nice to think of possible solution to remove pain but for now, the one which works is reducing the usage of the templates, with subst being the apprach which is probably best. Since this is driven by costs, if there's ever a removal of those costs the reasons for it will go away (the replication lag one is the toughest one to remove). Jamesday 15:24, 7 Apr 2005 (UTC)
I've linked to this page and discussion from quite a few relevant pages. Most people either don't work with templates, can't follow the technical aspects, or probably don't care. When Jamesday, the main database guy, says these are a problem, one has to be convinced. In fact, the stub sorting project is convinced and is moving away from using a meta-template. -- Netoholic @ 18:18, 2005 Mar 22 (UTC)
Although this page was mentioned on some of the pages where the people most likely to work on templates usually go, there probably should be another round of publicity before it moves from proposal to guideline (it certainly didn't receive as much publicity as Wikipedia:Requests for de-adminship). All the usual places, such as Wikipedia:Templates for deletion should get a new notice, plus the stub-sorting Wikiproject, plus some of the general Wikipedia info pages such as the Village Pump and maybe even the mailing list. You also should probably put a note on the Talk page of Users who have participated in this discussion so far (such as User:AllyUnion) in case they don't have this page on their Watchlist. BlankVerse ∅ 03:03, 23 Mar 2005 (UTC)
There's nothing wrong with doing things temporaraily, knowing that later we'll want to address the issues making it a pain. I'll certainly change my thoughts on this if the underlying issues change. Make it clear that it's driven by the current costs only, and can go away when those costs (replication lag the most intractable one) go away. Jamesday 15:24, 7 Apr 2005 (UTC)
I for one oppose moving this to a guideline. It's a good page to have, but these things should be decided on a case-by-case basis. — Itai (f&t) 17:31, 23 Mar 2005 (UTC)

I think that grendel|khan has a very good question above under #Question about caching. which should be answered first. If there really is a problem, then what could be done when a template is changed is not mark any pages that use it as dirty and just wait for them to get an edit and be rebuild using the newest version of the template. MarSch 17:11, 26 Mar 2005 (UTC)

I agree that technical fixes should be tried before enforcing rules like this.
If we did that, MarSch, we would then have to go and edit every article that used that template by hand to update them to the latest version, using up a LOT more time and bandwidth than having it update automatically. - Omegatron 19:31, Mar 26, 2005 (UTC)
No, the technical approaches shouldn't be tried first because those will take too long to happen and the performance is suffering today. Temporary expedients, doing things we don't want to do long term, work as a way of reducing pain. There may actually be NO even theoretically possible technical solution to the replication lag issue, though I've already asked for everything I can think of to reduce it and hope to see all of those in place in MediaWiki 1.5. Those will make it have reduced effect (by making each update faster) but can't make it go away. Jamesday 15:24, 7 Apr 2005 (UTC)

For a fair representation, perhaps all the templates which are considered as meta-templates should be listed here first? -- AllyUnion (talk) 06:07, 30 Mar 2005 (UTC)

Good idea. - Omegatron 06:18, Mar 30, 2005 (UTC)
Is anyone seriously interested in a list of meta-templates? One I've created is template:Prefecture navobox which is now used by roughly 800 articles on cities, and districts in Japan. Other high runner templates I've created or that I'm aware of include:
-- Rick Block 04:58, 5 Apr 2005 (UTC)

Have all technical fixes been tried? One possibility I don't see mentioned here (which might be a lot of work, but if the problem is actually severe it might be worthwhile) would be to automatically convert template references and template content to javascript allowing templates to be cached independently of the page that references them. If it's not clear, the idea is to convert a reference to a template T to HTML somewhat like: 

 <script language="javascript" src="T.js"></script>

and translate the contents of the template to the appropriate javascript code stored at the T.js URL. The net result of this is that a change to the template does NOT need to set the dirty bit for any referencing article since the browser would fetch the javascript independently of fetching the article text. This would tend to increase the number of URL fetches, but a change to a widely used template would have very little performance impact. I don't think mediawiki currently requires javascript to be enabled when browsing - this would require javascript. I'm not as familiar with stylesheets, but I suspect something similar could be done with stylesheets (without requiring javascript). In either case, this might be limited to templates that have no parameters. Another possibility would be to augment squid with the ability to understand a server-side include mechanism. -- Rick Block 23:34, 4 Apr 2005 (UTC)

Rebuttal

Disagree

I disagree with this opinion piece; I disagree with anything that further colors it with the tint of policy, guideline, or any other official or semi-official position. It appears to me to be a mission of the author, and questions have been raised that suggest interested parties have been prevented from improving it to appeal to a wider group of WPdians. I feel it errs in attempting to adapt Man to the Machine, rather than the Machine to Humanity. — Xiong (talk) 07:15, 2005 Mar 31 (UTC)

Perhaps I'm biased; I just finished creating a set of metatemplates myself. But I think it might be just as honest to say that the template work I did and my position on this article stem from the same deeply-held conviction:

Human time is more valuable than computer time.

This is the main current of thought among programmers and developers since the first great box full of vacuum tubes was brought to life: Let George do it. The first computers were programmed by connecting components, one to the next, with dozens of patch cords. Then, somebody decided that the computer could store instructions, and execute them -- one after another. The machine could be wired *once*, and once only; thereafter, all the programming would be done in "software".

Early computers were programmed in machine language: raw numbers. This was, and is, both extremely difficult and prone to error. In the right hands, it was blindingly efficient, too. In time, special programs allowed programmers to write in assembly language, using mnemonics -- short, vaguely readable "words" -- mixed in with the numbers. This was less efficient for the machine to execute, but any loss was quickly made up by advances in hardware -- bigger, faster, cheaper. Meanwhile, programmer productivity went way up.

Assembly language gave way to higher-level languages; perhaps the most powerful common language in use today is C++. This is an extremely sophisticated language, in which programmers write in something very like a highly-stylized English. They do not even always say what the machine will do next; instead, they merely describe objects, which interact with each other as they are defined. This makes far more complex programs possible -- that is, it makes it possible for humans to write them. Once written, a C++ program is compiled into a rather inefficient machine language, which is all any computer can actually execute directly.

Fortunately, we now have computers powerful enough both to compile C++ and execute the resulting inefficient code pretty quickly. Our resources have grown to permit us to do more advanced work.

HTML and similar markup is another development of the same idea. If you want a computer to display text on the screen with a box around it, there are far more efficient ways to do this than rendering some HTML markup in a browser. But the advantage of HTML is that relatively inexperienced users can master it, while even the most basic C++ skills are a bit difficult for some to grasp.

Wikimarkup is still another step in the same direction: Let George do it. Before Wikimarkup can be rendered in a browser, the wiki server must parse it into HTML. The HTML is interpreted by your browser, which was probably written in C++; converted into machine-readable data by inefficient (and buggy) code, and finally displayed on screen. From a technical POV, this sucks. But Wikimarkup means your grandma can program a computer, even if chocolate chips are all she thinks about when someone says "cookies".

Templates are a very clever, flexible way to standardize Wikimarkup, so pages all over a wiki have a consistent look-and-feel. They aren't the only way to do this; but they're accessible to everyone. They do incur all kinds of overhead -- servers must work hard to render lots of pages with lots of templates and lots of templates that use other templates. But the payoff is the same as it has always been: More users are able to do more powerful things better, faster, more easily. Let George do it.

There has always been a countercurrent of industry insiders who, by virtue of their superior competency, are able to do more with less, and loudly demand that others do the same. We must resist this anti-trend. We need to keep moving in the right direction. We can always go back to wiring the computer up for each job; we will get very fast execution on small, slow machines. Or we can continue building larger, faster, better machines that allow us to use them more easily. This is Good.

Specific points:

Yes, bigger-better-faster servers cost more Cash Money. More efficient, capable, reliable servers demand that humans maintain them and their code.

I keep hearing people say that Wikimedia Foundation is swimming in money (which I don't believe). I put forth a small effort to enhance donations and it fell on deaf ears. ("It ain't broke, don't fix it.") Well, it can't be both ways. Either there is a shortage of money, and suggestions for getting more merit attention -- OR -- there is a surplus of money, and we can go out and buy more and better machines and hire wonks to work them. I gather that every scrap of MediaWiki code has, up til now, been written by volunteers. I agree with the ideology -- and, yes, the code should remain free-speech-free -- but I see no harm in paying some kids to do actual work-for-hire. Everyone involved in this project (I have to think) has a Real Job and can only do so much for free.

Yes, much template stuff could be done in CSS style sheets. In fact, a lot of it uses CSS locally, and might be overridden by users.

But I do not see any way to edit the Monobook style sheet, let alone the Cologne Blue style sheet. I can edit my copy, but the WP-wide default is read-only. And, quite frankly, I'm not 100% sure I want those pages open to user edits. And it is anti-wiki to suggest that we get together and beg Somebody to make the changes for us. Things do not really get done around here by forming consensus first, then making the change. It's more like change-talk-change-talk-change-talk-change=consensus.

Yes, widely-used templates are vectors for all sorts of attacks. Yes, we could slam this door shut on all users, thus keeping evildoers out as well. While we're at it, we should restrict all editing to logged-in users, and perhaps demand email addresses too, so we can check up on returning, banned vandals and their sock puppets. If we didn't have so many ignorant, pimply pre-teens editing articles way above their heads, we'd surely have higher-quality content -- so perhaps we should verify users' credentials before allowing them to edit related content. We'll call the project NuNupedia.

Or maybe we can continue to deal with all sorts of attacks, case by case.

Yes, we can draw up detailed guidelines and formats for everything under the sun. We can even hope editors will read these guidelines before editing. We might even require editors to read and obey them all. In fact, we could impose a strict hierarchy of punishments to be levied on those who create non-standard edits. Example: User creates new article with pluralized title. Penalty: Ten lashes in public square.

Yes, {{subst:sometemplate}} answers a lot of performance and security problems. But it does so by gutting the power of the template technique in the first place. Since a template included using subst: is "frozen" at the moment of edit, nobody can automatically change it on all pages where it's used -- not for evil, nor for good. I believe that Good is more powerful than Evil. Good thrives on Freedom; Evil thrives on Secrecy and Control.

There are times when it is absolutely correct to use subst: and I agree it is underused; I even admit I've failed to use it properly at times. I especially support the use of subst: when choosing a licensing template for uploaded images -- after all, what does it mean if someone later (quite properly) changes the text of the license?

Templates that help to support a standard look, feel, and work across a large project are just Good Ideas. This proposal to suppress them does not move us along the Wiki Way.

— Xiong (talk) 07:15, 2005 Mar 31 (UTC)

Nice theory. Now tell me how to get rid of the problem today and for the next six months. Please stop doing things to make something you know is already a problem worse. Yes, I agree nicer solutions will be good but we don't have them today and today is when we have both the problem and a workaround which helps to reduce it. For now, subst will do the job. Not perfect but imperfect beats piling more load on an already stressed area to see when it breaks (it already has - search is now mostly turned off, which is what happens when there's just too much database load to handle). We're also getting three more powerful database servers... but they still aren't around today and I and all of us need to deal with the present, not the things we hope for (though we certainly can ask for improvements to make that future more the way we want it!). The short-term technical expedient which is likely to work is adding a switch to turn off template expansion and flipping that switch, causing template text values to be displayed instead of template contents. I'm just putting off asking for it in the hope that the less unpleasant things will happen instead. For now, search is gone and it's things like those discussed here which might cut load enough to get it back.

The Wikimedia Foundation has money and we're spending it. The things it buys do take time to arrive and temporary measures work until then. To give some idea, the new database servers are likely to be dual Opteron CPUs, one with 16GB of RAM, two with 8GB. One with 18 15,000 RPM SCSI drives, the other two with 18 10,000 RPM SCSI drives. Yes, that's really eighteen drives, to get 18 drives worth of disk seeks.

The current set of 5 are described on meta: 4 or 8GB of RAM and 6 drives each, various types (15,000 RPM SCSI, 7,200RPM SATA, 10,000RPM SCSI, 10,000 RPM SCSI, 10,000 RPM SATA). Jamesday 15:24, 7 Apr 2005 (UTC)

That's absurd, you're basing this semi-policy/guideline/whatever _exclusively_ on minor temporary technical issues. So essentially, what you have is a temporary guideline that very few people will see by the time the problem is solved on the technical side, both via hardware and template caching. This is just alarmist nonsense, wikipedia is not going to explode because of meta-templates existing while you wait for new hardware. I'm not even sure why you're using the time it takes to arrive as an argument anyway. We don't use the pony express anymore, when you order something it arrives very quickly. We have these new fangled things called trucks and airplanes to deliver mail.
All I keep hearing is constant whining from developers and admins about server load all because they were too lazy to implement fixes that were proposed MONTHS or even YEARS ago. I say this guideline is patent nonsense and the developers and admins actually become more responsible, rearrange their priorities and implement the necessary infrastructure changes instead of wasting time on other stuff and whining when they themselves don't implement stuff that they've been given a long time to implement. "Hey guys, I'm going to work on this new neat-o feature on mediawiki rather than working on more important stuff and dick around for the rest of the day instead of working on high priority issues." Oh and before you reply with the "buuuuut youuu dont know meeee :(( :( :(" argument, I do know and have been following this irresponsbiel behavior for a while. You guys are like a bloated, ineffecient government organization that has all these proposals that people ignore and other less necessary stuff gets worked instead. I suggest you go take a look at all the corporations running large scale, high bandwidth websites that don't have anywhere near as many problems as wikipedia has on a regular basis. -Nathan J. Yoder 17:54, 11 May 2005 (UTC)

Worst cases

Used on the calender sources then used on the calendar themselves! Triple include! -- AllyUnion (talk) 06:48, 30 Mar 2005 (UTC)

Don't forget the 30 day ones too. -- AllyUnion (talk) 06:48, 30 Mar 2005 (UTC)

Add Template:2005Calendar. -- AllyUnion (talk) 06:53, 30 Mar 2005 (UTC)

Template:Message box - glorified stylesheet. -- Netoholic @ 07:24, 2005 Mar 30 (UTC)

How are any of the above bad? They are not going to change very often and the user is not going to see the source. --mav 09:33, 7 Apr 2005 (UTC)

For each template, a fetch from the database is required, right? So if you have a template within a template, the server will need to attempt to retrieve the data two times before correctly displaying the information. Here, we have a case of templates being applied three or four times, requiring the server to make several trips to retrieve the data. Add it to a normal article, and you add more trip. Which increases the server load. -- AllyUnion (talk) 08:47, 9 Apr 2005 (UTC)

Requested move

  • This is a personal opinion piece by one user, and should be acknowledged as such.
I am suggesting that it goes into user space as User:Netoholic/Avoid using meta-templates, or into Meta as m:Avoid using meta-templates. Susvolans (pigs can fly) 12:31, 31 Mar 2005 (UTC)
    • It's an issue for all projects, so meta is a good place far a copy of it. Still needs to be on en for en implementation and to make it easier to find for en users. Jamesday 15:24, 7 Apr 2005 (UTC)
  • Give me a break and stop with the controversy. This isn't an "opinion piece", it is a description of how meta-templates work from a system side. The vast majority of the text of this page was verbatim from User:Jamesday - THE database admin. Other people have edited this page, and its been cited often. -- Netoholic @ 17:27, 2005 Mar 31 (UTC)
    • Sadly, "THE database admin" doesn't actually understand, at his own admittance, how certain types of caching with memcached work under the current mediawiki implementation. -Nathan J. Yoder 17:58, 11 May 2005 (UTC)
  • Agree -- I support the requested move. This is an opinion piece. — Xiong (talk) 17:31, 2005 Apr 1 (UTC)
  • Disagree. Every policy or guideline starts off as an opinion piece, albeit one you notice more than one Wikipedian shares. Instead of exiling this to user space ASAP to avoid having to discuss it any further, I suggest convincing Netoholic, Jamesday and any others who agree that they're wrong instead. This may sound shocking, but I think it's part of building consensus. :-)
    While this is still clearly marked a proposal, it's not harming anyone now, is it? I would obviously oppose any promotion to semi-guideline or true policy as well, but I see no reason to kill it.
    In fact, why am I still arguing? The mere fact that I still want to discuss it should be enough to prevent any move from happening. Hahaa, I too have the power of veto! (I'm kidding, people. Really. :-) JRM 00:57, 2005 Apr 2 (UTC)
    • People are going to be entirely unable to persuade me to lie about server performance problems.:) I have to live with reality as it is, not as I'd like it to be. That includes making decisions like turning off search because I couldn't persuade people to cut load in other ways, like that discussed here). I can ask people to cut load, just as electricity companies do. If they don't then the blackouts follow because the power available has to match the demands on it. Jamesday 15:24, 7 Apr 2005 (UTC)
  • Also disagree. Dan100 10:29, Apr 3, 2005 (UTC)
  • I'm changing my vote; I overreacted. This opinion might mature into viable policy Someday. But it is opinion, not a mere, bald statement of technical fact. Netoholic should be restrained from shoving his opinion through without that necessary maturation, and must allow other editors to contribute to its development. Someone who does not have a dog in this fight should refactor both article and its talk to remove inflammatory statements that appear to give this opinion more weight than it has in fact acquired. — Xiong (talk) 05:42, 2005 Apr 2 (UTC)
    • It's a bald statement of technical facts and Netoholic's responses to do something about those technical facts. Whether you or I like those technical facts doesn't change their existence. Jamesday 15:24, 7 Apr 2005 (UTC)
  • I disagree that this should be moved; just rebutted. - Omegatron 20:20, Apr 3, 2005 (UTC)

It was requested that this article be renamed but there was no consensus for it to be moved. violet/riga (t) 09:03, 4 Apr 2005 (UTC)

Why this advice is wrong

A nice polemic section to get the edits flowing.

Let's separate fact from opinion, first. I think we can give Jamesday his credit and take his description of the page building process for granted. Not that I'm truly interested in all the details, mind you; those are fine for the talk page but the blurb as a whole is quite overwhelming. If James had just said "meta-templates are really bad for performance because every inclusion in X articles adds Y computrons to a retrieval" I would have simply taken it on faith as well. In that vein, the "vandalism would be really bad because they're so performance-intensive" argument is really not a separate argument, just a strengthening of the performance argument. We will deal with that shortly.

The "vandalism would be really bad because the changes would appear in a lot of articles" is a separate argument, but this is a case of "no good without the bad". This is exactly why many meta-templates exist in the first place, and as with anything for which vandalism is really bad (page moves, the Main Page) we have ways of limiting the damage (restrict to logged-in users, protect and obscure through templates). The keyword here is restrict, not disable. I'm sure that if vandalism of meta-templates becomes such a big problem (and I'm not aware it's one already) we will deal with that as we see fit, and I hardly think completely avoiding them is a relevant alternative in that case, unless very satisfactory replacements can be found. We will deal with those shortly as well.

Now, the opinions in this proposal, as I see them, are:

  • The performance burden is significant, and we cannot expect it to reduce even in the long term.
    Rebuttal. As people have suggested above, are we already out of technical solutions? Isn't there any way to speed up the process if you take the existence of meta-templates into account in the first place? Obviously a trade-off is involved: developers would like it a lot if users just stopped using them; users would like it a lot if developers sped up the processing. But users will only stop using meta-templates if developers can convince them that it's really very expensive, and there's no way to improve that: the software is there to support us, not the other way around.
    • That is not "opinion" unless you count us having to turn off search becasue the database servers are overloaded as opinion? Yes, we are out of technical solutions today. If we aren't in six months I'll change what I write in six months. What the developers will do is turn off earch to reduce server load. That has already happened. Then they will either turn off special pages and watchlists or add a feature to turn off template expansion or do whatever else is doable quickly to keep the sites working. I don't ask for changes like this unless I have significant operational problems to deal with and dislike the alternatives available. The software can be improved and is being improved. That doesn't solve the problem we have today. Jamesday 15:24, 7 Apr 2005 (UTC)
  • The existence of meta-templates encourages the creation of poorly conceived child templates based on them. Avoiding them will diminish their numbers, which is good.
    Rebuttal. Encouraging people to be smart by making things harder doesn't work. "Make something foolproof and nature will invent a better fool." Yes, people can create silly templates. They always could, and always will. We're talking a template—on-the-fly substitution. This is not so complicated that avoiding it will stump users with a bad idea so much that it has a noticeable impact on the number of ill-conceived templates. But even if it did, ill-conceived templates can simply be removed. As far as I'm aware, we are not suffering an acute problem of being overwhelmed by bad templates proliferating on pages—no more so than WP:TFD is handling, that is. This smells strongly of personal revulsion to the phenomenon, which, while understandable, is not an argument for anything—not unless it can be converted in objective criticism.
  • The alternatives offered are adequate.
    Rebuttal. What others have already said: they're not.
    Design, document, and implement
    A way of doing meta-templates without using meta-templates: instead of editing the proposed template, you edit a regular page. Instead of using the talk page of that template for central discussion and Wikipedia:Template messages for documentation, you use a regular page. Instead of using "what links here", you maintain this list manually on a regular page.[2] In short: separating the actual template from its implementation. This is make-belief neatness, as it's equivalent to an integrated approach but without actual integration.
    Make use of CSS
    Avoid globals. It's unworkable to propose that any attempt at a uniform appearance should involve a change to the global stylesheet. The threshold for innovation would be unreasonably high. Replacing well-established meta-templates only used for appearance with a CSS class may be an attractive optimization and is certainly something to think about, but suggesting the global stylesheet as the only way of going for uniform appearance is iffy at best.
    Use lists, not templates and categories
    This idea has already been discredited by others. Lists are high overhead work, and most of it is automatable. So let it be automated. The argument that you could combine other tasks with lists is like saying that sorting different-sized ball bearings by hand is preferable to doing it by machine because it gives you the opportunity to check them for defects while you're at it. There's no reason to assume there are so many naturally combinable tasks that you're really no worse off without machine assistance—and even if this were so, this just forces you to combine those tasks even if you don't want to, because the overhead is prohibitive otherwise.
    Fix the problem, don't mark it with a template
    Good advice. It just doesn't have anything to do with meta-templates in particular. This is something to consider before using any template. Proposing that only fixing is allowed, and not identifying the problem, however, is distinctly counterproductive—and in some cases impossible. If I see a POV article, can I just "fix it" so it's NPOV? Most probably not, because my "fixes" will draw disagreement from others. Fixing problems is a good thing; the ability to clearly identify them for others is also a positive good, one that should not be lightly discarded.
    Use {{subst: }} when creating the daughter templates.
    From the argument itself: "...which probably defeats the point of using a meta-template in the first place". Exactly. Obviously, in places where we don't need updatable meta-templates, we shouldn't use updated meta-templates. Unfortunately this will rarely be so, and suggesting it should be is not really an alternative.
    The point is standard text. Subst is one of the ways of using standard text. You only need dynamically updating standard text if it's necessary to have the most recent version and too hard to change it in each place where it is use. Meta templates are one of the cases where it's quite easy to change it in each place it is used.
  1. ^ I am aware of the problem "what links here" is suffering for meta-templates, as outlined above: instead of listing only its direct children, it lists all articles indirectly using it. This could be called a feature, but then we also need the feature that lists only direct children. This is an implementation issue, though, not an argument against meta-templates.

JRM 00:57, 2005 Apr 2 (UTC)

When I first put this page together, and each time I quote it, it is so that I can refer people to it without having to re-explain each and every time about the technical problems with meta-templates. I don't do it to imply this is already policy, just that it presents the reasoning behind my actions.
Frankly, I'm shocked that people are ignoring the one voice on this issue which cannot be dismissed. This page describes the very real technical concerns about using meta-templates. That description didn't come from me, it came from User:Jamesday who is the project database administrator. I really don't care if thirty editors come on to this page to disagree... when someone so close to the technical issues has spoken out, there is hardly room for dispute. This guy knows how the system works, has done his best to make things work better, and can still see and describe the very real impact of wide-spread meta-template usage. I am sorry if I take his words and elevate them above fellow editors, but I just cannot bring myself to ignore him. I myself have surely created meta-templates in the past, but upon looking into this and reading his description, I have stopped, reversed what I could, and have worked to get this out to as many people as I can. -- Netoholic @ 18:27, 2005 Apr 5 (UTC)
Why isn't he here fighting for it, then? - Omegatron 18:30, Apr 5, 2005 (UTC)
James is also in the process of changing job, home, country and related things. He doesn't have time for this. I might have a chance to look again in six weeks or so. Jamesday 15:36, 7 Apr 2005 (UTC)
First of all, he took the time and wrote for a us a very nice, long technical description - what more do you want or need? He does have a talk page after all, if youwant to ask questions. Second, he's probably a bit busy lately keeping this site online with rapidly-growing traffic. -- Netoholic @ 18:35, 2005 Apr 5 (UTC)
This is silly. If Jamesday's concerns are really so pressing, why doesn't he just announce on the mailinglist or the Village Pump that we are in serious trouble, and if we would be so kind to restrict use of (meta-)templates to an absolute minimum? Indeed, why build guidelines or policy like this at all? Last time I checked, the developers' word was still law. If we don't have a choice, then don't make it appear like one.
Jamesday said, quote, "better not to use meta templates if relatively few templates are involved and there's much concern for efficiency and page loading times". Thanks for the safety tip! Now can someone please explain to me why reasons that have nothing to do with efficiency and loading times are included, like "scope creep"? Where is the restriction to "relatively few" templates only? As it stands the proposal doesn't just say "think carefully before using meta-templates", it says "avoid them", period. So all this "shock" about "ignoring" the voice of wisdom seems out of proportion, and firmly stating that "I really don't care if thirty editors come on to this page to disagree" is likewise nonsense. I advise making the developers simply speak out on this once and for all, instead of having us act as their proxies to interpret the hints. Jamesday's technical description gives no indication of how badly the server load is affected in terms of how big a load we can handle. If we have no way of estimating the effects of a feature, it's silly to argue for its complete abolishment.
Again: if Jamesday had said: "I'd like you to stop using meta-templates altogether from now on, and look for other ways to do things" than that would have been that. But that's not what happened: Jamesday gave a detailed technical description of why meta-templates are a significant burden to the server. That shows they should not be used frivolously, which is good to know. This was subsequently built on to read "step away from that meta-template with your hands in the air—here are some poor alternatives". I think we're being a bit overzealous. JRM 19:44, 2005 Apr 5 (UTC)
Update: I've sent Jamesday a mail to ask him to clear up this matter in more decisive terms. We're close to starting religious wars over interpreting "divine revelation" here. Better ask the source directly. (Of course, Jamesday is quite busy, so it may take a while before we have an answer.) JRM 19:58, 2005 Apr 5 (UTC)
Divine revelation: use subst for meta templates used in many other templates (or used on many different pages) before we're forced to do more than turn off search. We can revisit this as we get better solutions - for now we need to deal with the present. When I started this, search was turned on. It's not any more - load grew enough that we had to turn it off. I don't have to like it, just do someting about it. I will, so will the others. We currently have switches available to disable: watchlists, special pages. Let me know which switch you want us to throw next (it will probably be special pages or a new one hacked in to turn off category display, which is now more load than special pages). If a template or meta template only affects 10 pages which are rarely changed I simply don't care about it because it's not used enough to matter. It's the ones used in hundreds or thousands of pages which hurt. Please also see [3] and [4] where I said it wasn't worth doing anything about a meta template which wasn't painful enough to matter. It's a balance question and using subst when there's no great need for the latest version can help significantly, without hurting significantly. Jamesday 15:24, 7 Apr 2005 (UTC)
Of course, when putting together this page, I removed a lot of the non-descriptive text from his posts. If you want to read where he has asked us not to use meta-templates (or any template on a large number of pages), see Template talk:Sisterproject#Technical impact of templates like this and his other posts on that page. I found them quite clear on the direction he's asking us to go - away from using templates on more than a small percentage of pages and away from meta-templates specifically since they exacerbate the problem. -- Netoholic @ 20:06, 2005 Apr 5 (UTC)
We're going to use templates on more than "a small percentage of pages". That's a fact. If we shouldn't use them then they should be removed from the software. They are convenient and obvious solutions to the problem of wanting a similar set of information on more than one page. Are we sure that:
  1. The alternative, individual sets of text that are independent from each other and need to be edited all by hand separately would really use less server resources? (Are templates really making the wiki more efficient, and making it easier to add info, which is the actual cause of increased workload?)
  2. There is NO way to increase the efficiency of templates and meta-templates by altering the software? I haven't seen an answer to this question yet. - Omegatron 21:03, Apr 5, 2005 (UTC)
I've been looking into this a bit, and edge side includes (see http://www.esi.org/ to start with) looks quite promising (to me). I've posted a comment on User:Jamesday's talk page about this. -- Rick Block 23:53, 5 Apr 2005 (UTC)

A similar idea

I have gone a step further and created a new proposal that I would like to move to guideline status. See Wikipedia:Avoid using wikilinks. - Omegatron 20:21, Apr 3, 2005 (UTC)

Why a more prominent notice of non-policyhood is desired

Netoholic does not agree with the explicit stipulation on the page that there is no consensus for the proposal, describing the previous notice ("this is proposed policy") as "this notice clearly show it is not policy, while not "shutting it down" completely. There is still much support for this."

People have hammered on a more prominent notice that this is not merely proposed, but quite explicitly not policy. First of all, if there is "much support" for it, I haven't seen much evidence for that. I've counted plenty of opposers, but the supporters are remarkably unwilling to come here and talk about their objections. To top that, Netoholic has been referring to this article in justifying edits to meta-templates (this is my understanding, I haven't been involved in it), without making it clear to others that this is the situation. It all looks very authoritative when its decisive power is exactly zero.

It is disingenuous to say the least of continuously referring people to a "proposed policy" that is widely seen as problematic, without making any attempt at building consensus for it. This just gives a convenient excuse for not ever having to discuss changes on other pages, after all, you only have to refer them to this one to explain where you're coming from, right?

Stop pushing us, Neto. We know your opinion. You're entitled to it. Just don't go over our backs to promote it. The Wikipedia namespace is not a launching pad for your actions.

And everyone please stop this childish reverting. This is not getting us anywhere. JRM 17:21, 2005 Apr 5 (UTC)

I object only to the assertion that the request for guideline status "failed" after less than two weeks, and more strongly object that it can never work. Before a couple people came to this talk page and voiced pages-long diatribes on this subject, we were making progress incorporating some good feedback. I actually wish the objectors who have not will to make this into a good guideline would take a break for a while. Why do people have to live in the WikiNow by trying to cut this off at the knees? I have seen many people agree with the reasoning on this page, and apply it, though they are not as vocal as I'd hoped. -- Netoholic @ 17:35, 2005 Apr 5 (UTC)
Turning this around, why do people have to live in the WikiNow by trying this to justify decisions others soundly disagree with? Your point is well-taken, however. Let's not be overly pessimistic. I've weakened the note to meet both goals: express clearly that this is not a guideline or policy, but don't imply it will never be. Would that be more acceptable?
There's no point in continuing a discussion if you feel that those who are not providing "good feedback" are giving you "pages-long diatribes" you simply cannot have any answer to. Take a break? From what? Discrediting the fundamental assumptions on the talk page? I've said all I can say on that, really, so there's nothing to take a break from.
I have seen many people agree with the reasoning on this page, and apply it, though they are not as vocal as I'd hoped.
You are under no obligation to answer those "diatribes" by "objectors who have not the will", but anecdotal evidence of many people agreeing with you doesn't exactly sway me either. They don't have to be outspoken, but can we just get a few people here to come and say, "I basically agree with this" and leave it at that? Or take a straw poll, for that matter? Put it up for RFC? Anything? Lots of people who agree but can't seem to care about making this an official guideline are not convicing.
I cannot speak for others, but I "have not the will to make this into a good guideline" because I basically disagree with it, top to bottom. I happen to think constructive criticism, as opposed to tweaking, is a contribution too, but even if it's not, I'm not going to apologize for not turning away my head and allowing a proposal to become a guideline I don't agree with—guidelines are supposed to represent best practices. I frankly disagree that avoiding meta-templates is a good practice, as the arguments put forth haven't been convincing. And I've seen lots of "objectors", most quite vocal, but the supporters have fallen uncomfortably silent. Surely, if the objections raised here are all wrong and completely misguided, we'd like to hear so. JRM 18:08, 2005 Apr 5 (UTC)
"To top that, Netoholic has been referring to this article in justifying edits to meta-templates (this is my understanding, I haven't been involved in it), without making it clear to others that this is the situation."
Yes, he has. See Template talk:prettytable. It should be more clear on the proposal page that this is definitely not a consensus viewpoint. He had me convinced that it was, though I didn't understand why, simply by saying it was going to be turned into a guideline. - Omegatron 18:19, Apr 5, 2005 (UTC)

Avoid changing meta-templates?

It seems to me that the arguments here don't actually argue against using meta-templates. What they argue against is changing meta-templates. When a meta-template changes, it's expensive. Fine. Changing something that appears on many pages is necessarily expensive for the server.

Consider the following options:

  1. Lots of stub templates include the metastub template
  2. Lots of stub templates subst the metastub template

Now suppose that I don't like the wording of the metastub template.

Option 1: I change it. The servers work hard for a while because it's included in a zillion pages so their cached versions are invalidated.

Option 2: I go and change each of the stub templates. The servers work hard for a while because I'm editing a whole bunch of pages. Then they work hard for a while because I've invalidated the cached versions of every page I've edited (exactly the same set of pages as above!)

So Option 2 is more work for both me and the server.

Perhaps I have misunderstood, and even without changing metatemplates, they add significantly to the server load? Otherwise I just don't see any performance advantage to taking them out.

This is not to say there aren't some awful metatemplates: the calendar metatemplates were pretty ugly (but I think that's less painful than making all the calendar templates just as ugly!). Problems with individual metatemplates are entirely possible, but when text is supposed to be the same, make it really the same. Metatemplates provide the same advantages as templates, as well as providing the same costs. --Andrew 02:40, Apr 6, 2005 (UTC)

I guess the difference is that on option 1 you are doing all the work on a single transaction while on option 2 you spread it over a number of transactions. The total work is the same, but the atomic parts of the work are smaller. Also, with option 2 you give the servers some time to recover after each change. --cesarb 02:53, 6 Apr 2005 (UTC)
Who cares if the calendar metatemplates are ugly - they are not meant to be read. What matters is that they will very, very rarely need to be changed (thus no real cache invalidation issue) and the result is very usable. The current implementation creates just one set of metatemplates that can be used to create any year's calendar (not sure if the 'Source' templates are still needed - that is where the table code was but was generalized and moved to the xxDayMonthStartingOnY format so that table code need not be replicated each year). --mav 09:58, 7 Apr 2005 (UTC)
This is another common misunderstanding of the problem. Meta-templates are not just an issue when one of them is edited, invalidating page caches. Beyond that, meta-templates cause avoidable overhead just in regular daily use. MediaWiki maintains links between the end page and all templates and meta-templates used on it. For example, April 7 is linked to Template:AprilCalendar, Template:AprilCalendar2005Source, Template:AprilCalendar2005, and Template:30DayCalendarStartingOnFriday. Refreshing those links is a persistent overhead every time April 7 is edited. Each one is also a vandalism target. -- Netoholic @ 14:50, 2005 Apr 7 (UTC)
On the other hand, the server needs to spend time handling individual edits, doing previews, and so forth. From the technical description, it sounds like the atomicness of the process is not very important, given the servers' willingness to serve up old pages rather than stall. The human cost, of course, is much greater or option 2; all else being equal, I'd rather the computer do the work. --Andrew 04:11, Apr 6, 2005 (UTC)
The developers think otherwise. I personally don't like it when Wikipedia is offline, and anything we can do to reduce the burden on the servers is a good thing. -- Netoholic @ 14:50, 2005 Apr 7 (UTC)
Updating isn't the only reason these are bad. As they sit, they use more resources than using separated templates. The solutions given here are to avoid using meta-templates, becuase using them creates avoidable overhead on many levels. -- Netoholic @ 03:01, 2005 Apr 6 (UTC)
What extra resources do they use? What avoidable overhead do they introduce? Not using them also has a significant human overhead; is the avoidable overhead greater? --Andrew 04:09, Apr 6, 2005 (UTC)
RTFP. -- Netoholic @ 05:41, 2005 Apr 6 (UTC)
Can you feel the love, people? - Omegatron 13:34, Apr 6, 2005 (UTC)
I did. Thanks for your kind response. The only extra cost I can see when the template does not change is in a mysterious paragraph, describing how when a page is edited, every page it links to is "refreshed". If that means "re-read from database", then this amounts to one extra database read for a meta-template (if no parsed templates are not cached). Is that all you meant about overhead? --Andrew 06:15, Apr 6, 2005 (UTC)
I have come to the realization that your proposals all come from a dramatic lack of understanding of the technical description. -- Netoholic @ 06:29, 2005 Apr 6 (UTC)
The total work in Option 2 is almost certainly more (perhaps far, far more work for the editor and likely at least twice as much overall work for the server), however the burst of work on the server given the current implementation of templates creates an acute problem. IMO a policy of "don't use meta-templates" is not a solution, since users can (and seem to want to) create meta-templates. It is fine advice as an expedient work-around ("doctor, it hurts when I do this"), but an actual solution involving some sort of software change should be found. One draconian change might be to disallow including templates within templates. I think we can do better (edge side includes are apparently supported in squid 3). I suspect wikipedia is approaching (or is already at) the point where commercial vendors might be interested in some of its performance issues. Is this an avenue anyone has pursued? -- Rick Block 04:45, 6 Apr 2005 (UTC)
Does it actually produce an acute problem? What happens? From the proposal, it looks like if you edited a metatemplate used in 18000 pages, there'd be a 90-second period where some users saw outdated pages. I can live with that, for the convenience of using templates in templates.
It seems to me that the key instruction is "Avoid editing widely-used templates", which applies whether or not they're meta. --Andrew 06:15, Apr 6, 2005 (UTC)
How acute is not having search any more? That's where we're at now because of general database load (the sort of load which happens the first time a page is viewed after being touched because of a template change or because of any other editing to the page). This may not remove that problem but it'll help. Jamesday 15:24, 7 Apr 2005 (UTC)
What is the point of having a meta-template that you never intend to change? Note that I've already captured this idea in the last line of Wikipedia:Avoid using meta-templates#Scope creep. -- Netoholic @ 06:18, 2005 Apr 6 (UTC)
Well, you could create it once, and then change it as rarely as possible - for example, the (horrible) month meta-templates never need to be changed. --Andrew 06:26, Apr 6, 2005 (UTC)
The acute problem could be remedied in software by using old copies for templates that are too widely used until the pages have time to update. Not ideal, but then neither is the 500-page limit on "what links here". And if busy templates are rarely used, this is rarely an issue. --Andrew 06:26, Apr 6, 2005 (UTC)

Option 2 is the least unpleasant option for the servers and those using the wiki while it's happening. It usually avoids the replication lag issue and is likely to usually be invisible, while option 1 isn't. Please do it using option 2.

You're right that the total work is less. But it's all grouped together in one big lump and that's unpleasant.

There are several issues. One is ongoing load. That's caused by any use of templates and worse for meta templates. That load is incurred any time any page using a template which uses a meta template is edited and any time any template it uses is changed (it's on each edit preview and the first view of the page after any change to he page or templates it uses). Search is now turned off because we have too much of this overall load (from all sources) for the servers to handle. Subst within the first level template helps here without adding an excessive amount of human editing (in part because bots can do it and in part because having a version which tracks every version change of most meta templates isn't necessary).

Then there are the issues (like that replication lag) which happen when the template and meta templates are edited. Those are more likely to be visible at the time but aren't the sort of thing which causes us to turn off search (except temporarily while it's happening). They are the harder problem to solve because it's not possibly to buy more servers to deal with them. Jamesday 15:24, 7 Apr 2005 (UTC)

The title is "Avoid"

I wish that the respondents on this page would step away from discussing things that have to do with overall performance. Honestly, it's getting to the point of being rhetoric, and this page isn't the place for it. Sure, we know all' templates have a cost, so does each wikilink, category, image... The point is that we have some really bad templates out there which can be made much better from many perspectives by applying this advice. A guideline expressing that we should avoid using meta-templates is something very good.

I want to clear up one thing people tend to miss. This page is "Avoid using meta-templates", -not- "Never use meta-templates". I can anticipate that there are some very few legitimate and un-avoidable uses. This page is meant to document solutions to these problem templates, not to make more work for everyone, and definitely not to completely ban meta-template use. At one point (way up), we were trying to make a list of some of the worst meta-template offenders. I like this, and would like to return the page to productive discussion. If we can use this talk page to perhaps post notices like that, we can gather the thoughts of editors who understand good template design and use. Can we at least agree that there are some very bad, cludgy, unsimple, and ugly template "schemes" out there? Can the opposers at least acknowledge that User:Jamesday has expressed concern about overuse of templates, specifically meta-templates, and agree that at some level we need to take action? -- Netoholic @ 06:18, 2005 Apr 6 (UTC)

Well, in the proposal, what he is quoted as saying is really a lot weaker than what you;re saying:
>>If a meta-template is used in only a few or perhaps a few dozen templates which are fundamentally unrelated, it's debatable whether the extra equipment costs are worth it, compared to the relatively modest work involved in updating the individual templates.
In other words, he only suggests avoiding them if the other templates are fundamentally unrelated (like, I would say, the message box template) and not used in very many templates (I would give the same advice for templates, really).
In a comment here, he also points out that each template used on a page adds a database retrieval every time the page is edited. Is this cost actually a problem? And are metatemplates a major factor in it? --Andrew 06:38, Apr 6, 2005 (UTC)
But yes, I can certainly concede that overuse of metatemplates is probably a mistake, very like the beginning-programmer habit of making a function for everything.
How about a Wikipedia:Template style guide? This could discourage metatemplates as much as consensus decides is appropriate, but its talk page could serve as a place to bring bad template design. --Andrew 06:38, Apr 6, 2005 (UTC)
I suspect many respondents are focusing on performance since the guideline says the problem with meta-templates is the performance impact on Wikipedia, including a possibility of a denial of service attack via vandalism. If templates (including meta-templates) were "free" (in a performance sense), would we even be talking about this? My guess is User:Jamesday doesn't give a rats ass about how "unstylish" templates might be and that the point of his concern is the performance. If the message is really "avoid meta-templates, because they're inherently evil [because of some non-performance related issues] and, by the way, reducing their usage will also help wikipedia's performance", someone should take a stab at rewriting the page so that it says this. And, if this is the point, Jamesday's opinion is not worth anything more than any other editor's. On the other hand, if the primary issue IS performance, don't be surprised that people want to talk about performance. My point is not to inflame, but to try to clarify what the issue is. I suspect the primary issue actually IS performance and would support a temporary guideline of the form "avoid use of meta-templates due to their performance impact", which then could be split into two topics: what to do NOW (given the current implementation of templates) and what we might be able to do to improve template (and meta-template) performance. -- Rick Block 14:59, 6 Apr 2005 (UTC)
The only reason I'm here is because it's a load problem. Both ongoing (search turned off) and people not seeing recent changes or having all page views delayed while a big set of pages is changed. I have been and will continue to ask for software changes which will make this less of an issue. But none of them are here yet. So, it's temporary expedient time. Or was, three months ago, when search was still turned on. Meta tamplates tend to have a poor cost-reward balance - relatively few page edits necessary to update each individual template, relatively many pages must load the primary and meta templates each time they are edited.
You're correct with your rats ass comment. Actually, ignoring performance, I think something like the MayCalendar template is technically very creative as a way of getting the job done and I really like the power of such things. I just cringe when I see it at present.:) The technical me just gets to deliver the bad news that handy tools aren't yet efficient enough to use as we'd like. Jamesday 15:24, 7 Apr 2005 (UTC)
Well actually, if any of you paid attention to the templates that were created for 2006 to 2025 by the month, they aren't metatemplates. See Template:MayCalendar2006Source for an example. -- AllyUnion (talk) 08:56, 9 Apr 2005 (UTC)

Templates for federal representatives by state

This disuccsion has been moved to Wikipedia talk:WikiProject U.S. states#template for U.S. federal representatives

There's an active debate about a template on TfD that lists all US Senators, see Wikipedia:Templates for deletion/Template:Current U.S. Senators. My suggestion is that the places this template is used would be better served by a template listing the senators and representatives from a given state, with links to the complete lists of senators and representatives. See, for example, Template:CO-FedRep (based on user:Tomf688's Template:MD-FedRep). I can replicate this template for each state, and have started doing this (see Template:AL-FedRep and Template:AK-FedRep), but this bothers me since any change to the meta-format (including the links to the full lists of senators and reps) of the template will end up requiring changing all 50 (separate) versions. IMO, this is an example where use of a meta-template is warranted. If I use a meta-template there will be 535 articles using one of the 50 derived templates. Should I go ahead and use a meta-template in this case? Another approach, avoiding the meta-template, would be to write a single template that is given a state name which then includes the list of senators and representatives for that state by constructing the name of and including state-specific templates with these lists. Would this be any better (sounds worse to me, since I believe with this construction changing any state's list would uncache all 535 references to the master template)? -- Rick Block 18:01, 10 Apr 2005 (UTC)

I'd hold off creating any more for right now. Seems like this needs some thought-out architecture before implementing. I see a couple possible workarounds:
  • Create a single template to be used on all congressman pages in the form {{StateFedRep|state=Maryland|Senators=[[Barbara Mikulski]] • [[Paul Sarbanes]]|Representatives=[[Roscoe Bartlett]] • [[Ben Cardin]] • [[Elijah Cummings]] • [[Wayne Gilchrest ]] • [[Steny Hoyer]] • [[Dutch Ruppersberger]] • [[Chris Van Hollen]] • [[Albert Wynn]]}}. Essentially this takes your proposed meta-template and makes it the single template for all congressman pages. Formatting changes are done centrally, but changes to the membership have to be done to all that state's congressmen pages. Since this is usually only once every two years, shouldn't be bad. While the list of members can be long in the parameters, you can make the change once, and then it's a simple copy and paste.
  • Create a single template for each state's key political figures - Governor, Lt. Governor, Senators, Reps, (more?). There will probably some drift among the fifty templates, but it's not something to worry about too much as each can be tailored for each state and be a bit more useful than just Congressmen, like in Template:AK-FedRep, for example.
I'm undecided which I prefer more. We may want to move this thread over to Wikipedia:WikiProject U.S. states. -- Netoholic @ 18:54, 2005 Apr 10 (UTC)
The first option (one template, with the parameters specified in each article) would require 535 edits every two years. Largely cut and paste, yes, but compared to 50 edits (plus additional edits for the folks who get voted out or in) I don't think this is a reasonable approach. I'd far rather have 50 independent templates than this one. Moving the thread to Wikipedia:WikiProject U.S. states is fine with me. -- Rick Block 19:42, 10 Apr 2005 (UTC)
To be honest, there is no way to avoid a lot of work whatever solution is done. Even with the 50 template approach, you still have to modify the articles of the incoming and outgoing people, plus you'd have to update each of the 50 templates to change simple items, like the naming conventions of the associated lists, better formatting, etc. This will probably be done far more often than once every two years, and would require 50 edits each time when you could just change the one central template. This is why I'm undecided which is better.
Let's take another look at this - Do we even need it? How useful would these Congressman nav-boxes be? Couldn't we just make a link to the U.S. Congressional Delegations from STATENAME article in the See also section? Sure we force readers to make one extra click for this information, but that's true for many other potential things. Does this meet the threshold of true usefulness? -- Netoholic @ 19:56, 2005 Apr 10 (UTC)
The TfD outcome for Wikipedia:Templates for deletion/Template:Current U.S. Senators is looking to me likely to end up keep because the dominant opinion seems to be that two clicks to navigate from one senator to another is just too much effort. Let's move this to Wikipedia:WikiProject U.S. states. -- Rick Block 20:09, 10 Apr 2005 (UTC)
I've started a thread on this at Wikipedia talk:WikiProject U.S. states#template for U.S. federal representatives. -- Rick Block 04:04, 11 Apr 2005 (UTC)

Not a technical issue

Why so much debate on this? We're going over the same ground over and over. Templates make the machine work harder. Complex templates make the machine work harder still. Everything you ask a machine to do makes it work harder. That's the idea.

  • Congrats to a certain user for deflecting the impact of my last carefully-reasoned rebuttal of this whole silly notion that we should try to avoid using the machine "too much". But sorry; that was a temporary solution.

Machines overworked? Buy more machines, plug them in and turn them on. Share the load.

Not enough bandwidth? Buy more bandwidth. Not enough storage? Buy more storage.

Developers overworked? Hire some developers, stop expecting to get all the work done for free. Jamesday bitching? Pay Jamesday some Cash Money. I guarantee his complaints about excessive transclusion will abate, though they may not go away.

Budget overextended? Get more money, sell more teeshirts, wash cars if we have to do that; if every user in the system kicked in a dollar (or other local unit of currency) we'd have plenty of cash.

File:DonateTN.png

Here's what not to do: Burn all the tools that make work easier for us -- the humans here. Let's not go back to passing notes on scraps of paper. That works, too, and is a fine model of collaborative editing. But I like wikis. And I'm willing to pay Real Cash Money for the privilege of using one.

This is not a technical issue. Debate all day the question of how much "work" the machine does when you do this. It's silly; because work can only be measured in terms of human labor, and that means money. A room full of servers busily transcluding templates is not doing any work at all; it costs a certain amount of money to put them there and keep them there, but that's all.

We are the same people who tie up expensive, high-end 3D graphics workstations for the prime purpose of seeing how many Hobgoblins we can knock off with our Wands of Dragon's Breath. If you really believe bus cycles are a natural, finite resource to be husbanded like Black Gold, then quit playing Minesweeper and run the SETI screensaver -- who knows, maybe the Martian Messiahs will come and fix all our problems for us.

This is not a technical issue. This is a case of one overworked volunteer sweating an underfunded server giving vent to his frustrations, and one user, starved for attention and respect demonstrating to the community that he really understands what that poor guy is facing in the rack room, and therefore is a Real Techie, too. This is a case of small folk thinking small, and hardworking people getting dragged into a Black Hole.

Just forget all of this foolishness, and go back to work. Really. — Xiong熊talk 02:22, 2005 Apr 14 (UTC)

hahaha! i agree with you on pretty much all points. keep in mind that this whole page is just about a temporary treatment, though. the solution is to get more machines and better software, and these things will happen in the future, as i understand it. templates and meta-templates are beautiful things, and we should take as much advantage of them in saving human labor in the future as we can. in the meantime, we will just use subst: a lot and try not to edit those blasted stub templates (before you know it they will attain consciousness!), to avoid losing our ability to use the other beautiful things the pedia has to offer. - Omegatron 00:02, Apr 16, 2005 (UTC)

Emergency measures

I reply to Jamesday's comment:

...Now tell me how to get rid of the problem today and for the next six months....

As an emergency measure, of course, we can make less use of transclusion. We can use substitution instead -- and indeed, there are many times when this is preferable, not only because of server performance problems.

There are many things we can do, as an emergency measure, to reduce server loads -- some of them fine ideas in their own right.

  • Stop edit warring, especially over large, complex pages.
  • Stop edit warring over templates. It is especially good to be very sure your change is a good one when you edit a popular template.
  • Stop revert warring, move warring, tag warring.
  • When you have a disagreement with another Wikipedian, at least try to discuss it with him. Don't act out your disagreement.
  • Stop tagging templates marked for deletion with {{tfd}}. Tag those pages on their associated Talk pages, and use the form {{subst:tfd}}
  • The same can be said for much tagging. Most tags will work just as well if they are substituted, and many will do just as good a job if they are substituted on the page's Talk.
  • Think before you use a template. It is not really necessary to be careful creating a template; but whenever you decide to transclude, know what you are doing and why.
  • Consider substitution at all times. If you just want to make use of template contents, use {{subst:sometemplatename}}. Once you save the page, the link to the template is not merely broken; it ceases to exist, and causes no performance issue. The other side of this is that if the template is edited, its substitution will not change. This can be good or bad.
  • Don't use images just to be cute. "An image is worth 1000 words." -- well, not only is its value 1000 times that of a word, so is its cost.
  • Lay off the Preview button. Think about what you are writing. Use an external editor, spellcheck there, and proof your remarks yourself before pasting them into the edit box and hitting Preview.
  • But use the Preview button before the Save button. Multiple edits to a page make work for both machines and humans.
  • Edit by section. It is a lot less wear and tear on the page, and easier for you, too. When adding a new section, I often find it helpful to simply add the section head, save the page, and then reopen only that new section.
  • Archive Talk. Archive Talk to history whenever possible; long pages are hard to render, and if a user's browser times out, he is encouraged to reload the page over and over. There is no need to create a new page; history already preserves everything, anyway. All you need to do is this simple, two-step procedure:
    • Blank the page entirely. I like to insert a comment, such as "Archival in progress." You must do this first, in order to create a history link.
    • Open page history and copy the link location of the Talk page in its previous state. Paste this URL into the new, blank Talk page as an external link (using single brackets). I like to also copy the datestamp of the archive.

Most of these are good ideas at all times. — Xiong熊talk 06:06, 2005 Apr 16 (UTC)

While some (definitely not all) of these are good suggestions, they have nothing whatever to do with this page. Please find another avenue to suggest overall performance improvements. -- Netoholic @ 05:54, 2005 Apr 16 (UTC)

Discussion of archival

I'll say it stronger than Omegatron - stop screwing around with the talk page. Your "summary" does not make up for the fact that you removed a substantial amount of recent discussion, replacing it with your own skewed interpretation of the discussion. -- Netoholic @ 05:40, 2005 Apr 16 (UTC)

Prior to archival, this page exceeded 100 Kb, with a large number of templates -- ironically, many of them the "worst offenders". The page simply did not fully load under normal circumstances before crapping out.
There must be something unusual about your setup; pages of that size always seem to load fine for me, and almost never die in the middle of them. Perhaps the servers time your connection out if it takes too long to load the page? (What speed modem are you using?) I could belive that. JamesDay or a developer would have to comment on that, I don't know enough about the software to say. Perhaps there's some knob they could tweak which would make the site allow longer for larger pages, which would make it more usable for people coming in from slower modems. Noel (talk) 20:40, 16 Apr 2005 (UTC)
Archival to history is a normal and reasonable way to manage very long Talk pages. It's the way I archive my Talk page; it's the way many senior Wikipedians archive their Talk pages; it's even the way Netoholic archives his Talk pages. (He fails to link to the appropriate history version, though, so it's a little hard to follow what he's deleted.) The old discussion is not "gone"; it is right where anyone who likes may go to see it -- every word. And new discussion is not closed; it is still entertained right here.
Now, you can get just as mad as you like, Netty, but you may find little support for keeping the entire gory mess all on one page from now until the end of time. At one time, there was support for the idea that this should be moved into your user space, where you could in fact dominate and control it. But you insisted it be left in Wikipediaspace. That means it is going to get edited just like any other page in the common area.
Please note that you have reverted this page three times. Each time, I not only have to restore the archive; I have to preserve additional comments -- which duty I fulfill painstakingly. You are just making work. I do not have to tell you what you act in violation of -- over and above common sense. Let it be. If your ideas are to have any hope whatever of seeing daylight, you will have to subject them to the participation of the community. — Xiong熊talk 06:07, 2005 Apr 16 (UTC)
xiong, let it be yourself. leave the discussion page the way it was. there is no need to "archive" anything. stop trying to censor things you don't like. - Omegatron 12:15, Apr 16, 2005 (UTC)

Wikipedia:How to archive a talk page says: Cutting and pasting the text to be archived into a new page is considered the best method for archiving a Talk page. As to why people prefer this, I can only speculate. We all do know about histories (which have always been with us. so it's not like this is a new feature that we couldn't use before because it didn't exist), but most people seem to prefer the "archiving using a separate page" method anyway, so there must be a good reason. Wikipedia:How to archive a talk page does mention Separate archive files have the advantage that they are indexed and can be searched, which is certainly one point. It's also easier to archive individual sections, once they are quiet, using that technique.

I looked through all the sections here, and archived to /Archive all those which hadn't received fresh comments in less than about a week. Alas, that was only two sections, both short. (That was because JamesDay dropped in about a week ago and left comments in most sections - hopefully a number of those, which were otherwise quiet, will be ready to archive in a week or so.) Noel (talk) 20:26, 16 Apr 2005 (UTC)

"I don't know why the others weren't willing to archive"
I was not willing to change the page because there was a more than 3-revert "archiving war" and I didn't want to contribute to it. I was trying to ask the original cause of the problem to take care of it himself, instead. Thanks for the help.
I agree that the history does pretty much provide every aspect one would want from an archive. I don't know why we don't always archive things Xiong's way, or modify the software to provide the same functionality and leave the "archive display" uneditable. Makes much more sense to me. I've made comments on archived pages by mistake several times. We should bring it up on Wikipedia talk:How to archive a talk page. - Omegatron 23:51, Apr 16, 2005 (UTC)

How long do we do this?

Since we have now realized that this is a temporary treatment for a (lack of) hardware issue, how do we know at which point we no longer have to do it? How temporary is it? - Omegatron 20:18, Apr 22, 2005 (UTC)

Ah, but that's the genius of my "Emergency measures" (above). They are generally good ideas at all times, and we never have to stop taking them.
On the other hand, the emergency will never be "over". This is a not-for-profit enterprise, therefore chronically short of cash, no matter what optimists may say. We may temporarily get ahead of expenses, but the maw will quickly gobble up any excess and we'll be short again. Healthy profit is one way to tell when a not-for-profit enterprise has strayed from its primary purpose. — Xiong熊talk 18:07, 2005 Apr 24 (UTC)
Mmm. The project page itself should probably include links near the top to a couple of diffs where Jamesday sets out the problems in detail. Metatemplates = severe lack of caching, which can be problematic when you have the proportion of reader users vs editor users Wikipedia has (quite a lot) - David Gerard 23:21, 27 Apr 2005 (UTC)
The information from Jamesday is fragmented into discussions in several different places, including Template talk:Sisterproject, his User page, and probably elsewhere. It would be nice to see his comments and discussions refactored into some sort of coherent whole. There are also a number of questions that have been asked here and elsewhere that I don't think have been fully answered, but they will need to be answered from one of the developers, or wait till Jamesday returns. I also think that this discussion needs should be expanded to cover all the problems/ramifications of using Templates on the Wikipedia, and not be just about the single problem of Meta-Templates. BlankVerse ∅ 06:32, 28 Apr 2005 (UTC)

let's wait until 1.5

I would not really consider any performance questions serious until MediaWiki 1.5 is deployed on cluster. This will change whole view how we treat our systems and how those do respond. When we know what 1.5 is all about, and what should be do next, we will be able to decide on such issues. For now I'd like to be as conservative as possible. Domas Mituzas 14:57, 2005 May 3 (UTC)

Conservative in which direction? ;-) - Omegatron 15:18, May 3, 2005 (UTC)
In the direction of mucking about with a little as possible in the spirit of WP:POINT (IMNSHO, obviously). HTH HAND --Phil | Talk 17:04, May 3, 2005 (UTC)
WTFDTAM? - Omegatron 18:49, May 3, 2005 (UTC)
IMNSHO, naturally :-) HTH HAND --Phil | Talk 08:51, May 4, 2005 (UTC)

I don't understand Domas, you're saying we should wait until an upgrade to a more effecient version instead of engaging in pure speculation about perforamance from people who don't understand the current implemntation to begin with? Nonsense! By all means, lets treat this exclusively as a hardware problem where hardware is delivered by snails! -Nathan J. Yoder 18:03, 11 May 2005 (UTC)

which policy tag?

Re the section above [proposed policy]. Shouldn't this be considered one of the old "rules to consider" rather than a "proposed policy" - left on article by User:Pcb21

I think "proposed policy" is the most accurate. - Omegatron 14:54, May 12, 2005 (UTC)
Policies are enforced, rules to consider are not. Given the history of this page, I don't think it will ever be enforced, but should still be considered. Hence, I respectfully disagree. Pcb21| Pete 15:01, 12 May 2005 (UTC)
That makes sense. Is there a template for that, though? - Omegatron 00:39, May 19, 2005 (UTC)
I've just rewritten the intro to emphasise that these things are actually severely problematic, and in the class of things that you shouldn't do just because you can. I've tried to do so in a descriptive rather than prescriptive manner - "The devs say metatemplates are the work of Satan" rather than "If you use a metatemplate you will go to Hell." I've also added the real solution for fans of metatemplates: improve MediaWiki such that they don't have these awful effects.
Essentially, that they cause problems is a technical fact. That's really not something that will change with a vote. As such, it's a STRONG guideline, really - David Gerard 23:13, 18 May 2005 (UTC)
Good. Hopefully that will help clear up some of this mess. Can you emphasize that this is a temporary measure? See Jamesday's comments above. - Omegatron 00:39, May 19, 2005 (UTC)
Yes, though you could have ;-) - David Gerard 01:28, 19 May 2005 (UTC)
Oh, I dare not. You have seen the history page, haven't you? - Omegatron 02:10, May 19, 2005 (UTC)
I went through the entire history of the page and the talk page as part of the arbitration case against Netoholic ... hence the finding of fact that he was arguably 100% technically correct, but came across so badly he actually convinced people otherwise :-( So me (someone else) coming in and writing it is probably a good idea, yes :-) - David Gerard 12:52, 19 May 2005 (UTC)
User:Jamesday was correct about the problems with templates and meta-templates, but User:Netoholic's possible solutions were debatable. For example, his editing out of the mention of using "Subst:" because he preferred cutting-and-pasting of boilerplate text. His focusing solely on meta-templates and not considering all the problems with templates (such as the problems with edit warring on popular templates). And then there was his demonization of the Stub-sorting Wikiproject. BlankVerse ∅ 14:02, 19 May 2005 (UTC)
I tried to add something to this effect a while ago [5], but it was (of course) blanked and revert warred.
You say the developers consider meta-templates to be a disaster. As part of the ArbCom there was supposed to be a "referral" of these issues to them. Are your comments related to that referral? Can you provide some evidence of their disapproval and the details of what is actually wrong?
"100% correct" about what? I thought editing templates frivolously was one of the big server hits (see Jamesday's comments above), yet Netoholic is still revert warring templates and meta-templates ([6] [7] [8]) and claiming he is acting on behalf of the developers. I would imagine his edits are pretty taxing on the servers... - Omegatron 13:35, May 19, 2005 (UTC)

Reread ALL of WikiMedia m:developers Jamesday's comments on the article page an on the talk page. It's all pretty clear to me, but I used to be a database and software programmer. Here's an oversimplification for you. There is a single database call for each standard template. For each meta-template (double-transclusion), there are two database calls. For all templates that are very popular, and especially for a meta-template that is very popular, this can mean a hit on database performance. Remember that the next time things are slow on the Wikipedia. Everyone who reverted to the double-transclusion versions of the various Sister templates was actually doing there very best to make the Wikipedia slower! That includes User:Susvolans, User:Itai, User:Firebug, and the possibly sockpuppeteer User:KingOfAllPaperboys.

On the other hand, Netoholic should ALSO reread Jamesday's comments because most of what Jamesday talked about was not Meta-templates, but ANY template. That means that his edit warring on the popular Sister Project templates (as well as everyone else who participated in those edit wars) was causing significant, noticable, measurable database lags (although of a fairly short-term nature). Dispite his fixation on the Sister Project templates, he should have told his mentors and let them handle things.

Quite frankly the whole bunch have behaved rather badly, and if I had admin powers I'd give them all a symbolic bitch-slap in the form of a 24 hr. time-out so they could all think about their behavior. BlankVerse ∅ 13:54, 19 May 2005 (UTC)

If you believe that double-transclusion, or any overuse of templates, is bad (as you say in the first paragraph) then you could have done something to help with the Sister project templates. I don't care about the specifics of this the WP:AUM page or the recommendations it ultimately makes -- what I care about is the practical impact that Itai/Firebug/etc.'s intentionally careless use of meta-templates had on our project. You, and anyone else that read this page and agreed that Jamesday's advice was good, should have done or said something to help. Without that help or vocal support, I was placed in a bad position of defending a principle which no reasonable person could deny. You should be ashamed that you can sit here, now, and preach about who made mistakes when you agreed and did nothing. -- Netoholic @ 14:15, 2005 May 19 (UTC)
After the first two reverts don't achieve your intended effect, you need to try a different tactic. Instead you revert more than 10 times in a row.
I don't understand how you can have this pointed out to you so many times and still maintain that your actions are blameless. - Omegatron 15:17, May 19, 2005 (UTC)
Meta-templates or edit wars—they're both wrong. I've done my one very brief edit war, and that was one too many. I refuse to get involved in any more. I'll let admins, mentors, and arbitrators sort out who the black hats are in this case. BlankVerse ∅ 15:05, 19 May 2005 (UTC)
I've felt the same way, but if everyone gets fed up and refuses to get involved, the problems will have free reign. - Omegatron 15:17, May 19, 2005 (UTC)

"Helper" Templates for "Script" Programming

MediaWiki has developed several extremely convenient templates for use in what I might call "Script" Template programming. They use the Wiki Template language to provide branching statements and other ways to make template statements optional.

The real power of these "Script" programming meta-templates is that you could have an optional parameter. Consider how optional lines in a Taxobox might be used. Maybe {{Taxobox|Subfamilia=|other_params=...}}, and then within your template, use the m:Template:ine (If Not Empty template) to display or not display the "Sub-family" line. This seems like a much easier option than the current scheme of Template:Taxobox begin, Template:Taxobox end, and a hundred optional templates in the middle, one of which is the Template:Taxobox subfamilia entry.

There is a discrete set of these meta-templates that constitute the "script" programming part (m:Template:call, m:Template:ine, m:Template:if equal, and a handful of others). If you know anything about script programming, you realize how much more powerful these are than other approaches to optional parameters.

Yes, they are meta-templates, but they are almost as basic as the [[link]] structure. There would be no need to ever change them once they are implemented. However, if editors start using these, a vandal could be disastrous. (Imagine redefining how an "if" statement works!)

So, the question, (and please direct me to the right forum if this is not correct), since these meta-templates mostly don't exist on Wikipedia, and they are very useful, these seem like great candidates for introduction into Wikipedia as

  1. Protected pages (prevents vandalism and huge server load when they are changed); or
  2. Built-in features of the Wiki software (cannot be changed, and does not load the server by making it fetch another page).

And yes, I am a computer scientist. Mrendo 15:52, 15 Jun 2005 (UTC)

PS. I accidentally posted this also to another Template talk page. It didn't seem to be the right place, but I hit "edit" in the wrong window.

All of the meta-templates you mentioned are basically kludges to cover conditions where a more flexible m:Extended template syntax would be useful. To use templates in the way you describe is incredibly non-intuitive, server resource intensive, and hurts the eyes. -- Netoholic @ 20:07, 2005 Jun 15 (UTC)
These workarounds for missing control structures contain types of obtuse coding that I haven't seen since primitive compilers thirty years ago. Not good nostalgia. (SEWilco 20:14, 15 Jun 2005 (UTC))
Not all editors of pages which include a template have to understand the inner workings of the template, they just have to follow usage instructions. However, extended template syntax would be better.--Patrick 12:44, 18 Jun 2005 (UTC)
It's worth remembering that wikitext is supposed to be easy to understand. While I can follow these sorts of things, I have reservations about magic whch is not readily understandable or modifiable by a high percentage of editors. There's a school of thought opining that the current complicated templates are already undesirably tough, even without moving closer to making them a Turing-complete programming language. Simple single layer text inclusion was the original intent. If something is truly required, it may be better to seek an imporovement in the wikitext markup language instead. Jamesday 15:19, 6 August 2005 (UTC)

Still a problem?

Do meta-templates still cause a significant resource hit in the current version of the Wiki software? Will they ever not be discouraged in the near or distant future? I'm working on a set of infobox templates and want to know if I should consider using them. android79 00:53, July 28, 2005 (UTC)

I don't know for sure, but my guess is that the answers are yes (still a resource hit) and no (will still be discouraged in at least the near future). If soneone more in the know than me doesn't respond within a few days, you might want to directly ask user:Jamesday. -- Rick Block (talk) 02:28, July 28, 2005 (UTC)
See the Mediawiki 1.5 technical review section below for updates on 1.5. Jamesday 15:10, 6 August 2005 (UTC)

Developer committee review

(This topic inserted after the fact, for those trying to judge the veracity and timeliness of this guideline).

The issue of meta-template impact was referred to the m:developer committee as part of an arbitration request, whose input gives weight to guide (see below). The original impulse for guideline creation (performance impact at that time) occurred under 1.4 Mediawiki, and was reduced by 1.5 improvements. The necessity for the guideline may further diminish with future improvements. -- LarryLACa 18:04, 14 November 2005 (UTC)

Mediawiki 1.5 technical review

I'm not yet ready to comment on how much difference Mediawiki 1.5 has made. FYI, Domas Mituzas is also one of the core technical team (uses the name dammit in IRC). These are some issues for it to possibly address:

  • Whether all the slaves become out of date during an update affecting many pages, causing all viewers to see out of date information for seconds or minutes. Adding more servers doesn't help because each new one also gets delayed. Changes in 1.5 should help. Don't yet know how much - 1.5 is still being tweaked. Any improvement will only increase the number of pages which can be updated before people notice; it won't remove the issue, just possibly mask it enough for the number of pages currently affected.
  • The impact of purging cached pages which reference a modified template. All purged pages end up having to be regenerated the next time the page is viewed.

If 1.5 makes enough difference to matter in any area, I'll write about it here. Jamesday 15:10, 6 August 2005 (UTC)

Areas where changes in 1.4 and 1.5 have helped:

  • Locks on records during page updates used to be held while all pages were being purged from the squid cache machines. This contributed significantly to lock wait timeout errors and slow page update times for updates to the same wiki, as one update to a template could delay many other edits which needed to queue behind it because they were also changing something it affected in the database. There have been several major improvements to how the squid purging is carried out, substantially reducing this problem. Work to reduce lock durations in other ways continues. Jamesday 15:10, 6 August 2005 (UTC)
Just to confirm - the major problem remains (assuming it remains) changing of a template that is used on a large number of pages. That is, it is not the existence so much as the altering of them, and that the "meta-" is a bit misleading - it isn't the layers of indirection so much as the sheer number of pages a template affects? Pcb21| Pete 15:37, 6 August 2005 (UTC)
This is all misleading. Pcb21 is right; it's incessant editing of frequently-transcluded template original text that incurs a relatively heavy load -- and that is true for any template; multiple transclusion merely intensifies the problem.
And one more time, multiple transclusion and metatemplates are unrelated. A metatemplate is a template that refers, discusses, or otherwise refers to other templates, such as {{tl}} or {{doctl}}. Examples of a doubly transcluded templates are {{divstylered}}, when called as a parameter during a transclusion of {{divbox}}; and {{tfd}}, when used stupidly to tag a template that is transcluded on other pages.
Neither of these techniques causes undue server load; I should think anyone can see there is little cost in properly documenting a template by using a metatemplate. If editing a popular template hits the server hard, then it also is a powerful act that would otherwise consume a great deal of manpower. If done stupidly, then it is a stupid act; if done wisely, for good reason, a wise act. Like so many things, multiple transclusion is a tool that has the potential for harm as well as good. It is foolish to focus on suppressing the tool instead of educating the user. — Xiong熊talk* 02:56, 2005 August 9 (UTC)
Your example of {{tl}} is one way you might use the term "meta-template", but not the only one, and not the one used here. From the main page - "Meta-templates as used in this article are those that are created and used to keep other templates in a standard format." Please don't confuse the issue by confusing that term. In this context, it refers to templates within templates, or exactly what you call "multiple transclusion". That practice continues to be both a technical issue for the servers, as well as generally being a poor design practice which few true benefits.
PCB21 is right in that frequent editing of any template that is used on a high number of pages is bad. Perhaps he should write Wikipedia:Avoid editing heavily-used templates. That particular issue is just one part of why meta-templates should be avoided. -- Netoholic @ 04:48, 9 August 2005 (UTC)
If I understand correctly that is the only signficant technical reason to avoid meta-templates. The other reasons are social (harder for the casual editor to understand etc).
Re starting another page, we already have Wikipedia:Transclusion_costs_and_benefits as well as this page. If some refactoring is required to help editors out, then I happy to do that.
To Xiong re the term "meta-template". Yes you are quite correct that this is in a sense incorrect terminology. However it is probably too ingrained to change now :). Pcb21| Pete 07:48, 9 August 2005 (UTC)
Yes, the biggest issues, the ones which are hard or impossible to really solve, particularly as the number of affected pages becomes large, relate to the number of pages changed. With fewer pages it's "only" a load issue requiring more hardware to support the sites. That's not exactly a good thing either, but at least it can be done. Removing the costs after flushing cached versions of pages can't be eliminated at all (except by not flushing them) and the replication lag issue can be reduced, but not eliminated. Jamesday 19:28, 12 August 2005 (UTC)
I would assume that besides changing of a template that is used on a large number of pages, which causes a noticable, although momentary effect upon the server load, that having a meta-template that is on many regularly edited webpages has a less obvious, but probably greater overall effect upon server load. For example, someone has been talking about using complex meta-templates for creating the stub templates for different countries (a similar approach apparently is already being used by the Spanish Wiki_. I think that a similar effect would probably be caused by meta-templates that are used multiple times on the same page (e.g. the meta-templates that are used to display flags on sports team rosters). BlankVerse ∅ 20:06, 12 August 2005 (UTC)
I'm the author of the above flag and country templates. My understanding is:
  • That the increased "load" comes at the time an individual article is saved, not at the time an altered template is saved.
There is clearly "load" involved when saving an altered template that is widely used, specifically every page that references the template (via any arbitrary level of transclusion) must be marked in the master database as "stale" and all the squid caches of all of these pages must be flushed. If there were no transclusion and the referencing pages were edited individually, I'd guess the net load would almost certainly be higher but it would be spread over a far longer real time interval. -- Rick Block (talk) 19:40, August 14, 2005 (UTC)
So changing the template array entry for the image of the flag of Egypt, {{Country flag alias Egypt}} ("Egypt_flag_300.png") would flush any articles using {{flag|Egypt}}, while changing {{flag}} would flush all articles using {{flag}}. (SEWilco 20:14, 14 August 2005 (UTC))
I don't think the software is actually as smart as this. Since a change to {{Country flag alias Egypt}} affects {{flag}} (when invoked as {{flag|Egypt}}), I think it may well be that changing {{Country flag alias Egypt}} flushes all articles using {{flag}}. Jamesday could likely answer this with certainty. -- Rick Block (talk) 21:58, August 14, 2005 (UTC)
That implies that changing any template used within any template using {{ed}} (t/l) would flush all articles using templates which invoke {{ed}} (t/l). (SEWilco 22:35, 14 August 2005 (UTC))
  • So the alterations are distributed across time based upon how quickly articles are updated, thus the more articles using a template the probability is greater that one such article will be updated during a time period.
Since editing a template flushes the referencing articles from the cache, they're reformatted on the next read (not the next write). There are some database records updated only on a write (what links here, for example) but any change to a template invalidates any cached copy of any referencing page. Changing an image included by a template doesn't change the template itself, so does not flush cached pages using the template. -- Rick Block (talk) 19:40, August 14, 2005 (UTC)
Because my previous understanding was based upon when the cache is flushed, here are two new statements of how I think it behaves:
  • So the more articles using a template the greater the probability that such articles will happen to be in cache. (SEWilco 20:14, 14 August 2005 (UTC))
  • Because the cache has to be told what to invalidate, the entire list of referencing articles has to be sent to the cache thus the larger that list the greater the overhead of invalidation. Even if none of the articles is in the cache the list has to be processed by the cache servers. (SEWilco 20:14, 14 August 2005 (UTC))
    • If an updated article refers to more than one updated template, the update overhead is less than if the article had been updated for each template update. This is not due to templating, but simply because the rest of the article processing is only being done a single time rather than once for each template update.
  • If the same text were in an article, processing the text would be the same, so the additional load is in checking template timestamps and the transclusions needed to insert the resulting text.
I think it's worse, since this additional load is incurred whenever the article is changed regardless of whether the change is to a portion generated from a template. My understanding is that the main cost in formatting an article is fetching the data from the database. If the cost of reformatting an article is X, reformatting an article with one template costs roughly 2X (whether the change is in the "article" itself or in the template). The cost of reformatting an article including either two templates or a template that in turn includes another template is roughly 3X, etc. If every article in wikipedia included one template, we'd need roughly twice the hardware to support changes (primarily the Apache servers) relative to if there were no templates. -- Rick Block (talk) 19:40, August 14, 2005 (UTC)
Yes, fetching the templates has overhead. I was referring to the loads due to template alteration. I was just observing that processing the text resulting from transclusion is the same as if the text were in the article whether the templates had been changed during that edit or not. (SEWilco 20:14, 14 August 2005 (UTC))
  • There is no additional overhead due to flushing caches because that only happens when an article has been edited, which is an edit which flushes the cache whether a template is involved or not.
I'm not sure what you mean by "additional", but editing a template flushes cached copies of all articles including the template. If there are 100 articles including a template 101 flushes happen when the template is changed. If each article is edited individually (and there were no template) there would be 100 total flushes spread over the real time interval required to make the changes. 100 instantaneous flushes is higher load than 100 individual flushes spread over 5-10 minutes. -- Rick Block (talk) 19:40, August 14, 2005 (UTC)
Because cache is flushed at the time of an edit, I struck the sentence based upon there being no such flushing. (SEWilco 20:14, 14 August 2005 (UTC))
    • If articles are "touch" edited specifically to update templated material, this of course is an additional load as a side effect of a template alteration. This is not done by MediaWiki and is caused by whoever is tickling the articles.
(SEWilco 20:09, 12 August 2005 (UTC))

Low-Priority Template space

One change which may reduce overhead would be to provide a way to have templates which do not cause a cache flush. This would be most useful for templates which do not change often. It has been mentioned that Eloquence recently proposed shared spaces for information such as the capital of a country.

  • Does it seem like a good idea to allow transclusion to a Commons namespace? For discussion purposes I'll call it Global:.
  • Does transclusion currently support interwiki references?
  • Would editing a transcluded entry in the Commons database trigger a cache flush?
  • Editing a Global: entry, for example to change the capital of a country, should cause no changes until the next edit. An editor should know that the country's main article should be tickled to update that obvious resource, and other less critical articles would be updated later.

(SEWilco)

  • Is Commons in a separate data base server with a slower data path, which would slow the processing of an edit on en: servers? Or would processing be faster due to DB overhead being on Commons servers? (SEWilco 01:35, 15 August 2005 (UTC))

Scope Creep refocus

I wanted to see how severe the impact of the meta-templates cited in the Scope creep section is. I looked in them. I can't find any what links here references. I assume they were subst'd away. Current examples need to be cited or the text reworded to reflect a historic (resolved) exposure. -- LarryLACa 06:11, 12 November 2005 (UTC)

Oh..After input from User:BlankVerse on my User_talk:LarryLACa page, re: inconsistent usage of meta-template in the Scope creep section, I have reworded Scope creep slightly to be consistent with the actual creepage phenomenon. Chg /meta-template/skeleton template/ in 2 places and define skeleton template as ~ one used by copy or subst to create another.Skeleton contribute to the creation of high-usage templates, but are not themselves meta-templates. This edit retains the insight of the scope creep section. - LarryLACa 17:33, 14 November 2005 (UTC)

Scope Creep Comments

I (LarryLACa) copied the talk below from my talk, as the subject of the conversation is most appropriately addressed here. The lead reply is by User: BlankVerse in response to the above.

For background on your question at Wikipedia talk:Avoid using meta-templates. That page was started by User:Netoholic. He added the m:scope creep section as a diatribe against the Stub-sorting WikiProject, which he has always characterized as "busywork". The templates mentioned are NOT used as meta-templates as described in the rest of that article, but as the starting point (or guideline) for creating new topic stub templates, so the template is always substituted (subst:) when the new topic stub is created.

My attempt to more accurately rewrite that section was just one of the edit wars for which Netoholic was sanctioned (see Wikipedia:Requests for arbitration/Netoholic 2 and especially the /Evidence page). My personal opinion is the "Scope creep" section should probably be completely removed since it doesn't really "fit" with the rest of the article.

As a side note, since you are in LA, you might be interested in the Southern California WikiProject. Please take a look at the project and see if there is anything that interests you. If you have any comments or questions, please contact me. BlankVerse 12:27, 12 November 2005 (UTC)

Thanks for the feedback. I tweaked the guideline page accordingly: chg /meta-template/skeleton template/ in 2 places in the scope creep section.
I've also updated the m:Help:Templates page for server load to mention the guideline and reflect immediate vs when used impacts in the load model example. -- LarryLACa 20:55, 14 November 2005 (UTC)
As I said above, I personally think that the scope creep section should be removed. It doesn't really fit with the rest of the arguments against meta-templates, they aren't really meta-templates in the sense that it is used in the rest of the article, and it doesn't even fit the description of m:scope creep IMHO. Iff it is kept, it needs to be renamed and heavily rewritten to fit the current reality of the use of the meta-stub templates by the Stub-sorting WikiProject. (see my comments on the talk page for some of my complaints.)
I tweaked the Effects intro text to distinguish direct server load effects from ancillary indir. effects, of which Vandalism (exposure) and 'Scope Creep' are examples. Agreed: 'Scope creep' is a relative misnonmer in the traditional sense. It's not 100% off-base either: meta-templates do contribute to the expansion of code/maintenance costs etc, which are classic 'Scope creep' factors. -- LarryLACa 17:59, 15 November 2005 (UTC)
As for meta-templates themselves, you should start looking through the template namespace for some of the really bad examples, such as some of the templates used to display country flags for sports team rosters, where there are templates calling multiple templates that call more templates!
Not looking for work. I'm trying to get some article work done, for which I needed templates. Just trying to foward the consensus clarity here. I'll create a ToDo section with your suggestions. -- LarryLACa 17:59, 15 November 2005 (UTC)


To Do

Update guideline text: Server Load

Another thing that needs to be rewritten on the WP:AUM page is the "Server load" section. None of the stub templates have called the metastub templates since around the time that page was created, so the example of Template:Canada-stub calling Template:MetaPicstub has not been a valid example for a very long time. BlankVerse 06:53, 15 November 2005 (UTC)

Template Cleanup

Another example of wasteful use of server resources by meta-templates are where the only thing the second template does is have the color parameter for a font or span tag (e.g. <font color={{color}}>). There are bunches of these templates that need to rewritten, with the second templates sent off to Templates for deletion. BlankVerse 06:53, 15 November 2005 (UTC)


Yep. Some of this usage may even be cited in the m:Help:Templates (translation) discussion, further promoting ineffective practices. Re: colors: I think the better approach is to define color abbreviations at the server level, e.g. r g b corresponding to the standard word labels. -- LarryLACa 17:59, 15 November 2005 (UTC)
Beware: the actual impacts are not as bad as they seem, due to caching on both the web and DB server sides. a) impact occurs only when building the page for the webserver cache, which for popular pages is seldom. b) once the (unresolved) template has been retrieved from the DB by the web-server, it's in the webserver space and subsequent usage (multiple translations) is. c) if the (unresolved) template is not in web-server cache, it may still be in the DB cache space. Bottomline: the impacts are more on the order of (expanded) text size (a measure of cpu task crunch) than template count (an IPC factor), where task crunch << IPC impacts. Cleaning out the trash may not have much real impact. Cleanup often has a scope-creep downside - things tend to get improved (bigger) whenever someone looks at them. —Preceding unsigned comment added by LarryLACa (talk • contribs)

Timeliness and Consensus History

The necessity for this guideline depends on the ability of Mediawiki infrastructure to deal with the impacts. Since we have inputs from the developers for 1.5, the current (11/05) version in use at wikipedia, this guideline is still timely (but almost 6 mos. old).

There is no obvious summary that consensus for this guideline was achieved. From this talk page it's hard to see if and when the community coallesced to a common POV, unless you consider existence of the guideline page proof by example.

The point being that anyone reviewing the policy for relevance to a particular usage is going to be hard pressed to discern the underlying principles and whether the guideline is still active.

Re: prinicples. I tweaked the guideline JamesDay text to distinguish (when used) reconstruction costs from immediate cache marking costs.

Re: policy still active. I guess the talk page will have to speak for itself and the main page. —Preceding unsigned comment added by LarryLACa (talk • contribs)

Scope creep section

I'm glad to see changes being made to the page and being kept up-to-date.

Is the scope creep section really appropriate, though? The whole point of using templates is that they save work in creating new content. So this is a benefit, not a "harmful effect". The creation of lots of stub templates is a problem for WikiProject Stub sorting, not here. — Omegatron 18:13, 15 November 2005 (UTC)

Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia_talk:Avoid_using_meta-templates/Archive_1&oldid=1083649844"