Cannabis Ruderalis

Content deleted Content added
Doc James (talk | contribs)
→‎11b. score articles on "sockiness": sorry SmartSE/Smartse & clarify how machine learning works.
Line 226: Line 226:
::::::{{Ping|SmartSE}}: Agree that none of this was intended to imply automated reversion, only tagging/reporting with human followup. At most, automatic reversion of the article to [[Pending changes]] protection based on certain tripwires. Actually I think we could start on this last one right away with fairly obvious parameters, like "new corp article by new editor". — [[User:Brianhe|Brianhe]] ([[User talk:Brianhe|talk]]) 18:34, 2 September 2015 (UTC)
::::::{{Ping|SmartSE}}: Agree that none of this was intended to imply automated reversion, only tagging/reporting with human followup. At most, automatic reversion of the article to [[Pending changes]] protection based on certain tripwires. Actually I think we could start on this last one right away with fairly obvious parameters, like "new corp article by new editor". — [[User:Brianhe|Brianhe]] ([[User talk:Brianhe|talk]]) 18:34, 2 September 2015 (UTC)
:::::::You're pinging my doppelganger again ;) Does anyone have any clue who to ask about the feasibility of doing something like this? I noticed that neither of the original cluebot creators are really active anymore. [[User:Smartse|SmartSE]] ([[User talk:Smartse|talk]]) 20:50, 2 September 2015 (UTC)
:::::::You're pinging my doppelganger again ;) Does anyone have any clue who to ask about the feasibility of doing something like this? I noticed that neither of the original cluebot creators are really active anymore. [[User:Smartse|SmartSE]] ([[User talk:Smartse|talk]]) 20:50, 2 September 2015 (UTC)
::::::::Darn cut/paste. [[User:Brianhe|Brianhe]] ([[User talk:Brianhe|talk]]) 22:29, 2 September 2015 (UTC)


As an alternative to instituting a scoring system, consider using something like [[Amazon Web Services]]' Machine Learning to process the Wikimedia database, training it on known sock-posts, and letting it detect other candidates. Thanks to [[User:Smartse|SmartSE]] for the heads-up, BTW. (Declaration of interests: I'm not paid by Amazon or AWS, nor did they ask me to comment on this, but I do - as an independent - run the Cambridge (UK) AWS User Group.) [[User:Jonsg|Jonsg]] ([[User talk:Jonsg|talk]]) 22:23, 2 September 2015 (UTC)
As an alternative to instituting a scoring system, consider using something like [[Amazon Web Services]]' Machine Learning to process the Wikimedia database, training it on known sock-posts, and letting it detect other candidates. Thanks to [[User:Smartse|SmartSE]] for the heads-up, BTW. (Declaration of interests: I'm not paid by Amazon or AWS, nor did they ask me to comment on this, but I do - as an independent - run the Cambridge (UK) AWS User Group.) [[User:Jonsg|Jonsg]] ([[User talk:Jonsg|talk]]) 22:23, 2 September 2015 (UTC)
:Have you done machine learning before? I have. You don't just throw a bunch of stuff at it and wait for an answer. You have to give it two things. One is the dataset itself. The other is scoring algorithms (including a way to read/parse the data). The individual criteria for the scoring, listed above, are implemented by a programmer as algorithms, which the machine learning then finds the best weights for in the overall score, that give you the best [[Statistical classification|classification]]. Even for that, the definition of "best" is up to interpretation, because you have to decide what your acceptable false positive/false negative rate will be. — [[User:Brianhe|Brianhe]] ([[User talk:Brianhe|talk]]) 22:29, 2 September 2015 (UTC)


==12. Write a Wikipedia article about paid editing==
==12. Write a Wikipedia article about paid editing==

Revision as of 22:29, 2 September 2015

Ideas that may help address the issues of paid editing for discussion. Please feel free to add more ideas. My hope is to sit down with the legal department in Nov of 2015 to discuss possibilities in person.

Notification of this discussion

1. Ban some types of paid editing

Ban the type of paid editing that occurs via sites like Elance/Upworks. They have agree to automatically remove Wikipedia related jobs from their sites if we do this.

Discuss
  • I think this is definitely a good idea, as the handful of disclosed paid editors don't typically use those sites. As an additional point to this, perhaps we could establish a list of community endorsed paid editors to promote TOU compliance (added to a list by community consensus). In principal, I hate the idea of promoting paid editing at any level, and I'm not sure this is feasible, but it is important to give TOU compliant editors tools to compete against unethical non-TOU complaint editors. Winner 42 Talk to me! 05:18, 1 September 2015 (UTC)
  • This needs much better definition of what "the type of paid editing that occurs via sites like Elance/Upworks" means before anybody can seriously consider it. In theory at least there is no barrier to accepting a job at those sites and completing it while complying perfectly with the terms of use. Thryduulf (talk) 08:45, 1 September 2015 (UTC)
  • Yes so this proposal is to ban the type of paid editing that is funded through sites like Elance. There may be one or two people that edit properly through those sites. And yes we are sacrificing those one or two to deal with the thousands that are not following the TOU and have no intention to do so. Doc James (talk · contribs · email) 18:50, 1 September 2015 (UTC)
  • What is "the type of paid editing that is funded through sites like Elance" is the crux here. You can't ban something unless you can define it. Thryduulf (talk) 19:01, 1 September 2015 (UTC)
  • No that is not the "crux". The crux is are we in principle willing to ban this type of editing. If the majority oppose it no matter what the definition no need to spend our time arguing about definitions. Only once we have dealt with the real crux do we need to discuss the secondary issue of definition and for that I will get the legal team involved. That that is my question to you User:Thryduulf, do you oppose this type of paid editing? Doc James (talk · contribs · email) 00:48, 2 September 2015 (UTC)
  • I oppose all editing that does not follow en.wp policies, including undisclosed paid editing. I support all editing that does follow en.wp policies, including disclosed paid editing. Thryduulf (talk) 10:19, 2 September 2015 (UTC)
  • I'd imagine the rule would be some variation of "The use of online job soliciting sites (such as elance and upworks) to receive payment in return for editing Wikipedia is forbidden." Winner 42 Talk to me! 19:13, 1 September 2015 (UTC)
  • I don't think that would actually have the desired effect, as people would just find other venues that are less visible. There is obviously a market for paid editors, what we need to do is make it clear who is doing it according to the rules and make it easy for potential customers of paid editors to find out who is doing it legitimately and who isn't rather than drive everyone underground. Such a rule would also do nothing about the Orangemoody way of working. Thryduulf (talk) 19:21, 1 September 2015 (UTC)
This is about moving paid editing off of reputable sites. This is about making it more difficult. Yes it is not perfect for all types of paid editing but it is a piece of the solution. Doc James (talk · contribs · email) 00:00, 2 September 2015 (UTC)
I strongly disagree with you then - we want paid editors to be recruited through reputable sites where we can ensure that they are transparent and following our rules. A reputable site will work with us to ensure that our rules are followed and that those who do not cannot advertise through them, a non-reputable site will not care. Thryduulf (talk) 10:19, 2 September 2015 (UTC)
99% of paid editors purposely want to add advertising to Wikipedia as that is what they are being hired to do. No one is interested in disclosing who is paying them. The non reputable sites are hiring off of Elance.
How "can we ensure that they are transparent and following our rules" on Elance right now? Elance is not willing to engage with us in a very complicated manner or hand over the details of those on their site so we can double check them to determine if they are following our TOU. An seriously we do not hand the volunteer community to handle this. Doc James (talk · contribs · email) 20:33, 2 September 2015 (UTC)
  • If we are going to ban all paid editing then that needs to include via such sites. If we are going to continue to allow it within certain rules and codes then we want those sites to promote the types of paid editing that we allow. Allowing paid editing provided it goes through an organisation such as the one behind OrangeMoody and not if it is a direct relationship between client and legit transparent paid editor, would be the worst of both worlds. ϢereSpielChequers 12:00, 2 September 2015 (UTC)
  • The proposal is to allow "corporate accounts" that are allowed to comment on talk pages. Than disallow other paid promotional editing.
  • Upwork is willing to remove all Wikipedia related jobs automatically if we ban the type of paid editing that takes place through their site. This does not include WiR which we want but those do not work through Elance. It was their suggestion. Doc James (talk · contribs · email) 20:36, 2 September 2015 (UTC)

I support prohibiting the use of such sites. Of course a proper definition would have to be added but that should be easy. If people think that goes too far then we could require that the use of a paid recruitment site be disclosed as an affiliation of the paid editor. The proper place to do that would be at WP:PAYDISCLOSE

I'll add a subsection here for prohibiting most paid editing of BLPs, as it is also one of " some types of paid editing" Smallbones(smalltalk) 14:19, 2 September 2015 (UTC)

1a. Prohibit most paid edits to BLPs

It appears that there are many paid edits to BLPs. This not only distorts our contents, but sucks in naive individuals and helps support and encourage paid editing. Most importantly this rule would be very simple to explain "You can't pay to get your biography on Wikipedia." Simple explanations are very important since we have had problems getting the public to understand our rules. Proposed rule:

"Paid editing of biographies of living people is prohibited, except to remove content that is otherwise in violation of this policy. Content removal by paid editors must be noted on the article's talk page or at WP:BLPN. Paid editors may add information on BLP talk pages as long as their paid status is disclosed."

The proper place for this rule is at WP:BLP. Smallbones(smalltalk) 14:19, 2 September 2015 (UTC)

The same should apply to medical content. We should not have academics or companies hiring people to promote their new treatment.Doc James (talk · contribs · email) 20:40, 2 September 2015 (UTC)

2. Increase notability requirements

This is especially important in the areas of corporations and BLPs.

Discuss
  • I think this is more a matter of enforcing existing notability requirements rather than creating new ones (through increased NPP and AfD participation or similar). I'm not convinced that notability can or should be used as a tool to prevent paid editing. Winner 42 Talk to me! 05:20, 1 September 2015 (UTC)
Kindasortamaybe? Perhaps loosening the CSD for notability? The downside of that is more people are confused about how to create/have a Wikipedia article kept, which is often how people fall for PR schemes. I've thought about this for years and yet to see how this will change what is the overall status quo. We're popular, being on here makes you somebody, and nothing we can do (aside from failing) will change that. Keegan (talk) 06:10, 1 September 2015 (UTC)
Loosing the notability CSD criteria is something I am very, very strongly opposed to - A7 is already misapplied far more than probably any other criterion. Clarifying the notability criteria and possibly raising the notability requirements for corporations would probably help. The biggest thing that needs to happen though is more eyes on AfDs. Thryduulf (talk) 08:48, 1 September 2015 (UTC)
  • The problem in this case is that our existing notability requirements meant that deletionists could credibly threaten to get articles deleted that companies were willing to pay to try and keep up. How would increasing the proportion of articles that needed protection from deletionists reduce the risk of future deletionist scams like OrangeMoody? Clarifying the criteria, introducing some specific rules similar to the ones we have for sport or places would help, but I doubt we could get consensus for an "all companies that have ever directly employed more than y people at the same time" type rule. ϢereSpielChequers 13:06, 1 September 2015 (UTC)
  • There is an expression that I actually first heard outside of this project, "notable only on Wikipedia". Our notability criteria are far too low for both biographical articles and for businesses in particular, although it could also be applied to many other aspects of our project. Just as importantly, as Winner 42 points out, even when we have notability standards, people do not routinely apply them. It is rare to find an AfD with more than 5 participants, meaning that it is very easy to sway them toward a default keep/no consensus to delete. Anyone who has even the slightest desire to progress to adminship is very wary of tagging articles for CSD or any other form of deletion because any perceived error in judgment can often come to haunt them years down the road. Instead they use "may not be notable" tags, and we now have tens of thousands of articles with those tags whose actual notability is never really assessed or considered. Our low standards of notability have effectively turned sections of the English Wikipedia project into a cross between the Yellow Pages and Who's Who, both of which are fine for what they do but are simply directories for which the subjects pay for an entry. In many cases, we find that Company X located in Country Y (or entrepreneur X whose businesses are mainly in country Y) does not have an article on the "native language" Wikipedia because it would not meet that project's notability standards, but we'll have one on our project. This does not make sense at all. We need to (a) significantly increase notability standards, and (b) change our community consideration about 'enforcing' notability so that page curators and others do not have to "live down" recommendations for deletion on notability grounds. We should also go through all those articles tagged for notability concerns and either decide they're notable or that they're not; if they're notable, we should fix them and if they're not notable then we should delete them. We should aim to have no more than 0.005% of our articles tagged that way, and no article should make it through PROD or AfD with that tag still affixed. Risker (talk) 13:53, 1 September 2015 (UTC)
    I don't know where you find that editors are criticised for ancient errors in deletion tagging, but I can assure you that RFA at least doesn't work that way. We do get RFA candidates with a history of poor quality deletion tagging, but for that alone to derail an RFA the candidate needs to have made multiple recent mistakes. Isolated or ancient mistakes in deletion tagging are not on their own enough to stop an RFA. As for having no more than 0.005% of our articles tagged for notability, that simply isn't realistic. You would have to ration the deletion taggers to 34 notability tagged AFDs per day between them.... ϢereSpielChequers 18:21, 1 September 2015 (UTC)
  • I'd support this, but not for any reason to do with stopping paid editing - I just share Risker's concern that our bar is set too low with it comes to businesses, biographies and news, resulting in inherent problems with how we portray living people. In regard to paid editing it would make a dent (a lot of jobs are along the lines of "if my competitor has an article, why can't I"), but I'm not sure how much. - Bilby (talk) 07:33, 2 September 2015 (UTC)
  • Totally agree. The "two substantial references" for wp:corp is much too vague since the business press exists to fill its pages with content regardless of the actual importance of the information. I find the business press not unlike the music or film press -- these are essentially "fan zines" that will eventually cover every company/band/movie with an article or two. wp:corp needs to be tightened. As it is, there is an "arms race" for companies to get their information into WP because their competitors are there. This is especially the case for upstarts - I'm sure that Apple and Google are basically unconcerned about being included in WP because they ARE notable. I also assume that no one writes a WP article for a small company or a doctor's practice UNLESS they have a COI -- it's just not inherently interesting. I'd love to have a reason to reject or delete these articles. LaMona (talk) 16:49, 2 September 2015 (UTC)
  • Absolutely agree we need tighter notability requirements on business. It used to be that being in the Fortune 500 or traded on the NYSE was not sufficient notability. Now, according to The Independent "several wedding photography companies" were victims of the recent scam. Wedding photographers? Well perhaps if they are known for royal weddings and the like, but I don't see any need for "several wedding photography companies" in the entire encyclopedia. Three independent reliable sources that focus on the subject of the article should be a bare minimum. The single-store cafe on the corner might well come up with 3 reliable sources, especially if they advertise heavily in the local newspapers. Yes, I have seen an article on a 15 seat single-store cafe on Wikipedia. What we really need however is a bright-line rule. Perhaps sales of over $10,000,000 per year as reported in a reliable source or in a linked audited financial statement. That limit would likely let in a few hundred US restaurants - do we really need more than that? Smallbones(smalltalk) 17:09, 2 September 2015 (UTC)
  • Comment - agree in general, but it should be sufficient to enforce and clarify the general notability criteria for business topics. One re-occuring problem in particular: product announcements, feelgood interviews from branche-internal magazines and pre-reviews, before the product/artist/company are even really established, are often given too much weight. Such articles, often with little or no own neutrally researched content from the author, should not count towards our notability evaluation. They are often accepted with the argument "But it's a reliable source" - that argument is flawed: reliable sources are perfectly capable of writing promotional or other vanity fluff. GermanJoe (talk) 17:45, 2 September 2015 (UTC)
  • Agree Notability requirements need to be tightened and specific notability criteria need to be re-examined. Many of the specific criteria are fine until that one criteria like two book reviews or one professional game etc. and WP:NCORP while detailed is pretty useless becuase of the volume of PR press. These loopholes pretty much gut GNG. The community could start by defining the term significant as its interpratation varies wildly amongst editors and it a great point to wikilawyer when trying to sway an AfD.

    Further, no matter the topic, all articles should have reliable sources and an expanded BLPPROD should be implemented in all topic areas, regardless to when the article was created and with the initial sourcing requirement the same as the removal requirement ie RS not 'any-source'.

    This will help deal with the paid editors trying to push low-notability material into Wikipedia and will allow it to be removed easier. Paid editors on high-notability subjects are a different matter and likely a different population in general. The best solution is ban all paid editing, rollback all edits made by paid editors and delete all articles made by paid editors. I find it objectionable that so much volunteer time is used a) tracking down bad edits by paid editors, or b) improving articles of marginal notability, than we can not delete, created by paid editors thereby allowing the subject that hired them to get free work from us. JbhTalk 18:20, 2 September 2015 (UTC)

  • Agree with Risker. Andreas JN466 18:23, 2 September 2015 (UTC)

3. Increase requirements for new articles

Increase how long someone must be editing before they can create a new article without going through AfC. Maybe 6 months and 500 edits?

Discuss
  • Most articles in the Orangemoody case were actually taken from declined AfCs or Drafts. After that either a redirect was created that was then later populated by article text, or the article was recreated in sandbox and moved to article space. Technical measures around curating such pages would probably help better raising user requirements. Keegan (talk) 03:21, 1 September 2015 (UTC)
  • Yes good point they could just put the text at a redirect. Doc James (talk · contribs · email) 03:24, 1 September 2015 (UTC)
  • I would accept implementing the consensus at Wikipedia:Autoconfirmed article creation trial to run a trial to require autoconfirmed status to create or move articles into article space. (Please Doc, use your super WMF powers to make it happen) Ideally I think a 1 month, 100 edit requirement (and perhaps getting one or two articles through AfC could be an alternative way to gain article creation rights). This would prevent most speedy deletions in article space from occurring as that is sufficient experience to understand the basics of article writing, thereby increasing editor retention as new editors don't have their articles speedy deleted. Winner 42 Talk to me! 05:11, 1 September 2015 (UTC)
  • @Winner 42: I've dedicated the greatest portion of my wiki-career to helping new editors create articles, primarily in #wikipedia-en-help where I was a founder for several years, to my general CSD deletions. There is no amount of ACTRIAL or other confirmations that will deter this. Money is a far greater motive than meeting arbitrary software targets. Keegan (talk) 05:39, 1 September 2015 (UTC)
  • As this sockfarm was sufficiently organised to get their new accounts autoconfirmed, any extra measures we now take can forget ACTRIAL or any other solution based on the idea that only goodfaith editors stay long enough to get autoconfirmed. ϢereSpielChequers 11:18, 1 September 2015 (UTC)
  • The suggestion is to have different levels of "autoconfirmed" with a higher one for new articles. Doc James (talk · contribs · email) 18:52, 1 September 2015 (UTC)
  • Currently you don't need to be autoconfirmed to create new articles, and ACTRIAL, the proposal to limit new article creation to autoconfirmed users, was vetoed by the WMF. Since the OrangeMooody sockfarm was sufficiently organised to get accounts auto confirmed it is reasonable to assume that they would do what was necessary to get the new higher level of confirmation. Which raises the question about auto confirmation based proposals, what is the point of making things still more difficult for editors who don't use the services of organisations like OrangeMoody? ϢereSpielChequers 09:47, 2 September 2015 (UTC)
  • Unfortunately, I agree that any rules on editing longevity or numbers of edits will be thwarted by clever COIs. I think the only way to manage this is on real world criteria like references and notability, which are less manipulable than WP accounts. LaMona (talk) 16:53, 2 September 2015 (UTC)
  • Sure and CU can be gotten around by clever COIs. Are you suggesting we get rid of checkuser? These people are generally lazy and will try whatever to make money that is the easiest. If that is by scamming people via Wikipedia they will do it. This is why many people lock their cars and doors. Locks on your car or door will not stop a dedicated thief but no one would propose we outlay locks because it does not stop all thieves. Doc James (talk · contribs · email) 20:44, 2 September 2015 (UTC)

4. Create a new group of functionaries

Arbcom is clear that they do not see it as within their remit to enforce the TOU. We need a group of people who can receive concerns around paid editing that cannot be posted on wiki due to issues around privacy.

Discuss
  • I disagree. To me this is about overall juggling of titular responsibility. In practice the communities have no trouble enforcing their remits; it's arguments over remits that pass things to the WMF - or actual out-of-hands things - when a stalemate is reached. We'd rather have an arbitrary judgement than a stalemate. I'd rather have a stalement than an arbitrary judgement most of the time. Keegan (talk) 06:07, 1 September 2015 (UTC)
  • As has been repeatedly explained, the only group of people who can reliably make judgements about many TOU violators are the WMF as they are the only people with all of the time, tools, expertise and access to information to distinguish joe jobs from real humans. As an arbitrator who has been involved in trying to determine a connection between an off-site identity and a Wikipedia editor, it is not possible to do this with sufficient accuracy even with a group of people with arbitrator-level access. Thryduulf (talk) 08:53, 1 September 2015 (UTC)
  • Yes creation of this group would be closely done with the legal team. Doc James (talk · contribs · email) 18:48, 1 September 2015 (UTC)
  • This needs to be a meta level proposal so the group has the remit to tackle spam that runs across several languages. We also have the problem that Arbcom has historically been so busy that it burns members out, so spinning this out as a separate role makes sense. As for whether it should be run by staff or volunteers, first lets establish if the community wants this to happen, second we need to establish if we have volunteers to make it happen. If the community wants it to happen but can't get volunteers then we need to see if the community wants it enough to add to the list of things that volunteers want to have happen, aren't finding sufficient volunteers for, and therefore are prepared to pay people to do. ϢereSpielChequers 13:16, 1 September 2015 (UTC)
  • Yes agree will need to be a meta level proposal. Doc James (talk · contribs · email) 00:07, 2 September 2015 (UTC)
  • I hear arbcom members saying they can't do it, so obviously a new group of functionaries are needed. I don't see that this is much different from checkuser functions: private and incomplete data have to be analyzed and a decision has to be made if Wikipedia is going to keep on working. Somebody should take this to the WMF board if necessary. Smallbones(smalltalk) 17:28, 2 September 2015 (UTC)
  • Instead of volunteer functionaries the meta-discussion should propose the Foundation put together a small group of paid professionals to enforce the TOU. This is preferable from a legal standpoint as TOU enforcement is a legal question that has financial implications both for the subjects and the paid editors. If a volunteer team is used to suplement the professionals they need to be covered as agents of the Foundation. Anarchy is well and good but there must be dedicated and identified agents of the Foundation who can contact those entities who are violating the TOU and who can speak on behalf of the the Foundation - the entity with whom the TOU agreement is with. We, as editors, do not have the standing to enforce the TOU beyond the virtual space of Wikipedia itself and no real legal authority there. JbhTalk 18:32, 2 September 2015 (UTC)
I think it's fairly clear from this idea and others that technical solutions alone, volunteer community solutions alone, and legal solutions alone won't work. We need an integrated approach that uses the strengths of each set appropriately. The technical solutions could take some of the routine patrolling burden off of the volunteers. The volunteers can do what's appropriate for them and forward serious legal stuff to WMF Legal. Legal will have to do the black magic that only they can do. But this can form a virtuous circle where better machine- and volunteer-level stuff will give Legal more time to handle the things that really turn off the problem at its source. It's pretty clear to me at this point that the army of darkness that we're talking about fighting here is an organized, criminal-level enterprise and needs to be treated as such. In this regard I'd like to even go to "some next level shit" with US authorities, if WMF Legal hits a dead end. To me, the Orangemoody case looks like a RICO situation that they'd be interested in. — Brianhe (talk) 18:39, 2 September 2015 (UTC)
A number of those involved are in Indian and Pakistan and hired via Elance/Fiverr/Odesk. The US gov does not really have jurisdiction over were at least a part of the problem is coming from. Some of these paid accounts are working in 5 plus languages.
I agree this is going to need the community and foundation to work together. For the community to realize that we are here to write an encyclopedia not to try to create some utopian anonymous movement. That we need to adjust our perspective and rules to accomplish the new reality out their. Doc James (talk · contribs · email) 20:49, 2 September 2015 (UTC)
Online behavior has consequences
We shouldn't give up that easily because of non-US actor involvement, and I don't even agree that the US authorities are powerless. The services they are using to recruit overseas WP actors (Elance/Upwork) are US based and subject to its jurisdiction. There's a lot that the feds can do, once they decide to do it. Even overseas servers like The Pirate Bay and enterprises with no local nexus like Silk Road (marketplace) aren't beyond the reach of the law. I'm pretty sure that Elance didn't agree to help WP turn off the flow of recruitment of undisclosed paid editors out of the goodness of their heart. Nobody running a business wants to see records subpoenas, servers seized for evidence, or even domain takedowns (see image). — Brianhe (talk) 21:59, 2 September 2015 (UTC)

5. Allow some links on En WP

Allow the linking to Elance/Upwork accounts on Wikipedia. Currently it is not clear if this is allowed or not.

Discuss
  • We should not loosen WP:OUTING because of bad actors. Keegan (talk) 06:04, 1 September 2015 (UTC)
People currently link to Elance accounts. Most do not consider it outing. Thus this would not be a change to outing just a confirmation of current practice. For example
We should not allow the misuse interpretation of our guidelines to protect bad actors. Doc James (talk · contribs · email) 06:46, 1 September 2015 (UTC)
  • Loosing WP:OUTING is the very last thing we should be doing. It's OK to link in cases of self-disclosure on Wikipedia, and in cases where it is conclusively proven, e.g. by CU, that user:X here is user:X or user:Y there. It is never acceptable to out someone on the basis of suspicion or fishing. Thryduulf (talk) 09:02, 1 September 2015 (UTC)
  • I agree that loosening OUTING is not the way we want to go here. However, the community is currently between a rock and a hard place in cases of abusive paid editing: we can't discuss paid editing evidence onwiki, because outing, and we can't submit it to arbcom, because arbcom says paid editing is neither their job nor their problem (and that even if it were, they can't investigate things to their own satisfaction and thus couldn't do anything about it). The community is left with no real option for handling cases involving abusive paid editing. If we don't want exasperated people trying to handle it onwiki for lack of other options, then either Arbcom is going to have to start being willing to deal with these cases, or we are going to have to, as suggested above, consider creating a new group that is empowered to do it instead. A fluffernutter is a sandwich! (talk) 15:07, 1 September 2015 (UTC)
  • Outing does not disallow the posting of these links to Elance if you look at the examples . Doc James (talk · contribs · email) 18:55, 1 September 2015 (UTC)
  • I also don't see the point in this. If their editing is bad we will spot it (eventually) and if the fact that they are advertising on elance is crucial to an investigation it can be dealt with off-wiki. It would be a slippery slope to loosen OUTING. SmartSE (talk) 12:53, 2 September 2015 (UTC)

Posting links to other websites is NOT outing per se. Quoting WP:OUTING:

"Posting links to other accounts on other websites is allowable on a case-by-case basis."

....

"If ... personally identifying material is important to the COI discussion, then it should be emailed privately to an administrator or arbitrator – but not repeated on Wikipedia: it will be sufficient to say that the editor in question has a COI and the information has been emailed to the appropriate administrative authority. Issues involving private personal information (of anyone) could also be referred by email to a member of the functionaries team."

So it is not a question of whether we can report links to elance/upworks. The only question is how we can do this in the proper manner. I don't think it is completely clear who to report it to in order to have it acted upon. ArbCom has been very skittish on this - so where exactly should we report it? We can address that question. Smallbones(smalltalk) 14:55, 2 September 2015 (UTC)

The "case by case" wording was inserted without real consensus. There was a massive discussion a few months ago about it, and there was consensus against allowing it in any of the cases suggested. Speaking as an arbitrator, we do not have the time, skills or resources to carry out investigations of this nature. Thryduulf (talk) 15:42, 2 September 2015 (UTC)
There has to be somewhere to report it - otherwise you appear to be saying "we don't have a policy on paid editing" which is just not the case. Smallbones(smalltalk) 17:33, 2 September 2015 (UTC)
Then figure out which body of people does have the required time, skills and resources. Thryduulf (talk) 18:04, 2 September 2015 (UTC)
I've sent stuff in to functionaries and it seems like a black hole, sorry guys. What would really be better from my point of view is some kind of locked repository with a receipt that could be referred to. In other words, hypothetical example: I email my Elance link and evidence of real-world identity to a certain email address and get back a unique token in reply. Now I can refer to this token in a sockpuppet investigation or COIN case. Users with appropriate credentials can open the file by the token I supplied. Ideally there'd be transparency via audit trail as to who has shared what with whom. I don't see why the audit trail shouldn't be public, though maybe during an investigation it should not be. I think I'll develop the idea of a community investigation toolset a bit farther in a new subsection of this page. — Brianhe (talk) 18:23, 2 September 2015 (UTC)
As stated above User:Smartse our primary goal is not to create an anonymous group on the Internet it is to write a neutral encyclopedia. OUTING is not the be all and end all of everything. It is not a policy that we should try to interpret as widely as possible such that we can allow bad actors to do whatever they want and make encyclopedia writing impossible. OUTING does not cover linking to Elance and it never has. We do it all the time.
For example you want to disallow links like this [1]. What should we do about these sort of things? Yes arbcom wants nothing to do with paid editing. We get that. We need a new group of functionaries to do this work. Doc James (talk · contribs · email) 21:26, 2 September 2015 (UTC)
I don't see any problem with that link but some editors use their real names on elance and linking to those constitutes outing which always taken precedence. The problem is with the content rather than the contributor and if elance accounts claim to have written articles then we should certainly investigate them and clean up as necessary. That doesn't require us to link to the account though and if we were to, it opens up people to harassment. As much as I dislike undisclosed paid editing I don't think it is worth compromising RL identities unnecessarily. I think that pursuing the sites where ads are posted and persuading them to not allow the ads is a better idea as this would stop the problem at the source. I'll deal with your functionaries suggestion above. SmartSE (talk) 22:12, 2 September 2015 (UTC)
There is a possibly a real name in this link [2] now that someone has sign up for the job. User:Smartse do you plan to oversight it and take me to arbcom to get me banned?
The problem you have is that their is no consensus that linking to this type of site is outing even if real names are present.
If this is outing 100s of dedicated non paid WPs would be banned. Would leave Wikipedia to the paid socks which they would probably like. Doc James (talk · contribs · email) 22:25, 2 September 2015 (UTC)

6. Allow corporate accounts

These accounts are verified and have a corporate name attached to them. They are not allowed to directly edit articles. Allowing corporations a way to engage may prevent many of them from going to paid editors.

Discuss
  • I think this would have had more merit for en.wiki prior to the update ToU. Before that, following the German Model (flagging role accounts that represent people/companies is what the German Wikipedia does) could have been much more practical. Now it's far easier to register a PR account in your own name and list your employer, and just edit the talk page. The problem is with paid editors that do not choose to follow the ToU. This is the internet, there's no hard solution there. Keegan (talk) 06:03, 1 September 2015 (UTC)
  • I would allow accounts operated by individuals to include corporate names in their username, e.g. user:Brian (Notable Corporation) or user:Alison (Llanelli PR Agents Ltd). I don't like the idea of role accounts so much, as one employee should not be punished for the actions of a colleague. Thryduulf (talk) 09:09, 1 September 2015 (UTC)
    These accounts are not speaking on behalf of individuals. They are speaking on behalf of companies. As companies are people to they deserve their own accounts no? Doc James (talk · contribs · email) 18:57, 1 September 2015 (UTC)
    Companies are not people though. Brian is an employee of Notable Corporation who can speak on behalf of his employer, and part of his role is to manage the online presence of Notable Corporation. Brian should have an account that is officially linked to Notable Corporation. Catherine works in the same team for the same company, she should also have an count that is officially linked to Notable Corporation. This way Catherine doesn't get punished if Brian is unable to follow the rules. Thryduulf (talk) 19:10, 1 September 2015 (UTC)
    Yes wouldn't have issues with separate accounts. Doc James (talk · contribs · email) 00:09, 2 September 2015 (UTC)
  • "Corporate account" names are already allowed per WP:ISU--see bullet 4. --Izno (talk) 11:31, 2 September 2015 (UTC)
  • Allow "User:Brian (Notable Corporation)" but require an OTRS declaration to prevent joe jobs against Notable Corp., and prohibit them from editing article pages on their employer, competitors, and suppliers with a paid editing disclosure required. Smallbones(smalltalk) 17:42, 2 September 2015 (UTC)

See ongoing RfC here: Wikipedia_talk:Username_policy#RfC_-_should_we_allow_company_account_names_with_verification Andreas JN466 18:15, 2 September 2015 (UTC)

7. Delete articles by paid editors

Once an editor is confirmed to be a paid sockpuppet/undisclosed paid editor we should simply delete the articles they have created. We do not have the editor numbers to fix them all. And typically they are mostly of borderline notability.

Possibly, but only if they have not been adopted by genuine community editors. All their articles should certainly be examined to see if they meet any speedy criteria, and deleted if they do. Any criteria like this must apply only if the breach of the ToU was intentional. If someone was genuinely unaware of the ToU requirements, or tried and failed to meet them (e.g. because of misunderstanding) and complies when educated then their prior work should not be deleted unless it meets normal criteria - these are the people we want to avoid driving to undisclosed paid editing. Thryduulf (talk) 09:16, 1 September 2015 (UTC)
Yes exactly "only if they have not been adopted by genuine community editors". Most of them havn't as they are non notable topics. Doc James (talk · contribs · email) 18:56, 1 September 2015 (UTC)
Actually, there are quite a few Orangemoody article subjects who are or might be notable enough for an article. Not all paid editing is about non-notable subjects. Thryduulf (talk) 19:11, 1 September 2015 (UTC)
I did not say NONE of the paid editing is about notable subjects. I said MOST of it isn't. We have a list of articles here [3]. We could calculate just how many are notable. However to my eye the answer is not many. Doc James (talk · contribs · email) 00:52, 2 September 2015 (UTC)
Now you are conflating "articles created by Orangmoody socks" with "articles edited by paid editors", the first is only a subset of the latter. Thryduulf (talk) 10:22, 2 September 2015 (UTC)

This could be added to Criteria for Speedy Deletion, but the articles could be recreated without prejudice by other editors, or kept if other editors have made significant contributions. There is really no way to tell if somebody unintentionally violated ToU, so I'd just leave that out. Smallbones(smalltalk) 15:20, 2 September 2015 (UTC)

You find out if someone unintentionally violated the TOU by asking them. If they respond that yes they are paid but didn't realise they were breaking the TOU (either through unawareness or misunderstanding) and correct their mistake(s) going forward then the breach was not intentional and their work should not be deleted. I think any speedy criterion for this would have to be related to G5, which normally applies only to pages created after the user was blocked, but could (if there is consensus at WT:CSD for it) be applied retroactively if the user was banned for breaching the TOU regarding paid editing. Thryduulf (talk) 15:40, 2 September 2015 (UTC)

Yes - Simply remove the work of those who violate the TOU. After the removal they can go through a process to have the work un-deleted if they are good faith editors or truely did not understand what they were doing. Making deletion the default makes it much harder for AGF to be gamed. It is very easy for an account to say 'ooppss I'll never do that again' have their material kept, have some actual good faith editor improve the marginal material and accomplish their contract. They can then create another account and do it again, and again, and again. Maintaining 5000000+ articles is a non-trivial task we should take pains that paid editors do not increase that load by adding material that would never have been added but for the subject paying for it to be included here. JbhTalk 18:44, 2 September 2015 (UTC)

Yes - this has to be done. As of now the paid editors face insufficient cost. This would raise the cost of discovery quite a bit. Also it would disincentive them to play games at PROD or AfDs (like this, hello Orangemoody minions), which further raises costs on our side. It should be understood that a lot of the contracts stipulate that they only get paid if the article survives N days after placement. — Brianhe (talk) 18:53, 2 September 2015 (UTC)

8. CorpProd for businesses

BLPprod has raised the bar for BLPs, we could do something similar for businesses. A sticky prod that required "from this date all new articles on commercial businesses must have at least one reference to an independent reliable source" would give an uncontentious source it or it gets deleted deletion route for those articles that are only sourced to the business's site. ϢereSpielChequers 09:10, 1 September 2015 (UTC)

This is the best idea I've seen so far. It would need to disallow news stories that are basically a rewording of the press release and mentions in business directories, but that's detail that should be easily worked out. Thryduulf (talk) 09:18, 1 September 2015 (UTC)
  • I'd boost this to at least 3 references that consist of editorial content written by a reliable source, in which the article subject is the primary focus of the editorial content, and which is not mainly focused on "start-up" information. Lots of media that are generally considered "reliable sources" also allow articles that are not written by editorial staff on their sites (i.e., they allow "paid editing" by basically publishing press releases or similar 'reporting'), including just about every single business-related source. The vast majority of organizations that get start-up funding never actually get anywhere; perhaps include a requirement that the entity must have been formally registered as a business for a minimum of 2 years, and have significant reliable-source coverage of at least one successful profit-generating product/service before they are considered notable. Risker (talk) 14:06, 1 September 2015 (UTC)
  • Yes would be a start. Doc James (talk · contribs · email) 00:53, 2 September 2015 (UTC)
Strongly support Risker, i.e 3 reliable sources where the article subject is the primary focus. It may be difficult to clearly state her start-up criteria, but I support it in principle. There are the occasional start-ups (maybe 1 each year) that come out of nowhere with a groundbreaking product and are reported on internationally in dozens of sources, but probably those could be dealt with as the exception to the rule, e.g. via WP:IAR. Smallbones(smalltalk) 15:43, 2 September 2015 (UTC)
  • For corporate, there needs to be further qualification on what "substantial article" means and what "reliable source" means. Much of what comes out in the business press is just fandom - cooing over new products or lauding some company's success. I don't see those as being encyclopedic, just business as usual. What would we expect to see that would make a company worthy of an article? Personally, I'd go for some concept of social/historical impact, but I doubt if we can define that sufficiently. If we treated agriculture the way we treat business, we'd have an article for every farm and every year's new crop. LaMona (talk) 17:02, 2 September 2015 (UTC)
If we treated agriculture the way we treat business, we'd have an article for every farm and every year's new crop. that bears repeating. Thank you LaMona. Smallbones(smalltalk) 17:47, 2 September 2015 (UTC)
  • Support proposal by Risker. I thought the "multiple independent, reliable sources" part of WP:GNG kind of implied Risker's minimum of three editorially independent sources for corp articles. However, if we have to spell it out, I support that. — Brianhe (talk) 17:59, 2 September 2015 (UTC)
  • Agreed, per Risker. Andreas JN466 18:13, 2 September 2015 (UTC)
  • 'Agree per much of what I have said earlier on this page. Significant needs to mean significant as in covered, in detail, over a non-trivial time frame, in multiple reliable sources. If those sources can not be documented in some way when the article hits main space there should be a definite time after which it will be deleted. JbhTalk 18:49, 2 September 2015 (UTC)

9. Lower the bar for sockpuppetry investigations of spammers

We have a difficult balance between maintaining the privacy of our editors and investigating potentially dodgy edits. If "writing a spammy article on a borderline notable commercial entity" became grounds for a checkuser to check for sockpuppetry then we would be much more likely to catch rings like this in future. The vast majority of our editors would be unaffected by such a targeted change. ϢereSpielChequers 09:55, 1 September 2015 (UTC)

I would support. Doc James (talk · contribs · email) 00:10, 2 September 2015 (UTC)
Support Smallbones(smalltalk) 15:46, 2 September 2015 (UTC)
I support this notionally, but the devil's in the details. Getting consensus on what constitutes "spamming" might be difficult. And the problem with socking isn't usually strictly about spamlinks, though that can be part of it. Would the lower bar attach just to the spammer account (s) or does it spread to the whole investigation to which they are a party? The one way you encourage throwaway SPAs for spamming. The other way you have kind of an all-purpose tool for widening SPIs just by naming the right party. So I'm having trouble seing how this would be reduced to practice. Brianhe (talk) 16:21, 2 September 2015 (UTC)
  • Support I believe this to be one of the most important factors, along with making it harder to create multiple accounts and raising the bar for autoconfirmation. I think we take the "invasion of privacy" factor here much too seriously, since it is almost never the case that personal information becomes public through a CU investigation. The standard of required evidence should be loosened, long-term editors - especially those with a track record of sniffing out socks - should be given the benefit of the doubt, and SPI clerks should be more careful not to act as "agents of obstruction"; they should perceive their job as weeding out the wheat from the chaff, not -- as I think is the case now -- in denying endorsement unless the evidence is airtight. That kind of evidence doesn;t come along very often, but experienced editors do get a feel for socking, and that should be given weight which it is not given now. BMK (talk) 20:57, 2 September 2015 (UTC)
BMK has hit the nail on the head in his characterization of the SPI process. It's incredibly frustrating to bring a case there where I expect them to connect the dots, and have it fall flat on its face. Two cases in point to illustrate what I consider major process failures. (UPE=undisclosed paid editing):
  • Worthywords (talk · contribs · deleted contribs · logs · filter log · block user · block log): Created 22 January 2009. WP Ed instructor as of 2011 [4] Announced retirement from teaching Feb 2014. Started UPE 4 March 2014 [5]. SPI shows involvement in PE ring 1 July 2014 [6]. Disclosed under protest in July 2014 [7]. Renamed acct 3 September 2014 [8]. COIN 7 May 2015. Blocked 15 July 2015.
  • OnceaMetro (talk · contribs · deleted contribs · logs · filter log · block user · block log): Turned up on our radar from external investigation of Sony PR scandal and massive UPE/socking ~22 April 2015 (see Signpost special report). Blocked 24 April 2015. SPI filed 29 July 2015. Initially rejected with a snarky comment; spent hours adding evidence; some other accounts were confirmed, but as for what I think is the primary, "OnceaMetro is, unfortunately, Stale, so we can't run the CheckUser".
Not disparaging other volunteers, just saying the process needs work, yet frustrated. Look at the intervals between incindiary red flags and our responses in these two cases. Oh, other process problems above, if you didn't notice, the editor was granted a rename under a cloud, this is also true of Arr4 and someone else I can't recall right now. — Brianhe (talk) 22:22, 2 September 2015 (UTC)

10. Keep some IP data for longer

Currently we keep data about the IP's used by registered accounts for three months, this was a major limitation in this investigation. A general increase would be ill advised as it would risk exposing more of our editors to harassment via lawyers acting for spammers, but we could do a targeted extension, in particular for edits by badfaith accounts such as those blocked for sockpuppetry. That should make it easier to find returning bad faith editors. ϢereSpielChequers 10:03, 1 September 2015 (UTC)

I don't think this would be technically possible - either the data is available or it isn't. There is already a way for checkusers to record and store details of known sockmasters where this is desirable, however checkuser data has a high entropy - the older it is the less useful it is generally speaking (IP address get reassigned, people move, useragents change as people upgrade and switch to different versions of browsers, etc), so the benefits of having the data available for longer are not linnear. Thryduulf (talk) 10:45, 1 September 2015 (UTC)
It isn't technically possible to restore deleted data. But it is possible to keep existing data for longer, though this would require changes both to the software and in the legalese of the privacy policy. ϢereSpielChequers 11:13, 1 September 2015 (UTC)
I mean that it is not possible to do a "targeted extension", either there is a "general increase" or no increase. Thryduulf (talk) 12:39, 1 September 2015 (UTC)
I can appreciate that a general increase would be easier to implement, but the downside would be excessive in that all accounts would lose privacy and be more vulnerable to spurious court cases. A targeted extension would be more difficult to implement and would require careful thought as to who it targeted, but I don't see anything impossible in that. ϢereSpielChequers 12:45, 1 September 2015 (UTC)
How would it be targetted though? You don't know which accounts are sockpuppets/sockmasters until you check them so anything relevant would need to be discovered during the current timeframe or be lost, and there is already a way to record data before it expires where it is known this is needed. I don't see what benefits would be gained here? Thryduulf (talk) 13:09, 1 September 2015 (UTC)
Glad to hear we are retaining some data where there is due cause to be suspicious; I thought I remembered suggesting this during the review of the privacy policy. Perhaps the FAQ there could be changed to reflect what really happens. My suggestion still stands, though modified to "review the current exceptions with a view to broadening them" ϢereSpielChequers 13:41, 1 September 2015 (UTC)
  • One might think checkusers would be the ones most likely to agree with this; however, I for one think that 3 months is sufficient. Cases such as the Orangemoody one are rare and exceptional, and I think the WMF's limitation of 3 months (it's a global setting) is reasonable. It's actually longer than many other organizations as it is. As to the 3-month period being a limitation, that cuts both ways. It also meant we had a hard limit on how much checking we'd have to do with the OM case. If we'd had more CU data to review, we may have found more accounts (although their editing pattern is clear enough that most admins should be able to do duck blocks) but the investigation consumed a vast amount of CU and other functionary time. Investigations like this need to be rare, or we'll need a LOT more CUs. Risker (talk) 14:26, 1 September 2015 (UTC)
  • Support - This goes right along with the suggestion above. Once again, we are being much too sensitive to "privacy right" then we legally or morally need to be. BMK (talk) 20:58, 2 September 2015 (UTC)
  • We need to balance our real goal of building an encyclopedia against the ideals of privacy agree. Having a discuss between legal and CU on where the cut off on this issue should be I think would be good. Doc James (talk · contribs · email) 21:34, 2 September 2015 (UTC)

11. Actively look for mutual patrolling

One feature of the OrangeMoody sockfarm was an interlinked group of accounts that marked each others articles as patrolled. A computer program that watched for similar patterns in future and notified functionaries of patterns it had detected would give us a chance of finding similar farms faster. ϢereSpielChequers 10:20, 1 September 2015 (UTC)

This sounds like it would be a good idea. It would not be 100% reliable of course and there would be false positives and false negatives, but as a flag for human investigation it could work. I will have to leave it to others to say whether this is technically possible or not though. Thryduulf (talk) 10:47, 1 September 2015 (UTC)
  • We have to be cautious not to succumb to the "reds under the bed" syndrome. We've found a bunch of bad actors, who have leveraged our standard site procedures to their advantage and the disadvantage of article subjects/draft creators and the encyclopedia. Amongst the accounts were those that seemed to "specialize" in page curation, and were responsible for marking reviewed a pile of the articles, and most of those "article reviewers" didn't create articles. However, many of those articles were also reviewed by editors who do a lot of page curation who aren't socks here; they just do a lot of page curation, presumably because it's an activity they enjoy. It's certainly one that we need to keep doing. The interlinking of the page-curation socks and the article-creation socks comes from checkuser evidence to start with, which was then correlated with editing behavioural review; it would not necessarily be obvious without both types of data. Risker (talk) 15:08, 1 September 2015 (UTC)
    • But if a program was run and identified a group of editors who were patrolling each others articles or all patrolling the articles created by a second group of editors, wouldn't that be worth investigating by checkusers? ϢereSpielChequers 18:05, 1 September 2015 (UTC)
      • How would you identify the "article creation" group? Risker (talk) 18:13, 1 September 2015 (UTC)
        • By computer matching. If there is a ring of say 20 patrollers, half of whose patrols are random and half concentrate on the same thirty article creators, a large proportion of whose articles are patrolled by those twenty patrollers, then it should be easy for a computer to find that. ϢereSpielChequers 08:17, 2 September 2015 (UTC)

11a. Limit account creations per IP

  • Could we automatically check the IPs used to create new accounts and flag those that create many new accounts in a short period? The only legit occassion this would happen would be in the educational program. I wouldn't suggest blocking the creation, but if we could at least see the groups of accounts we could check them to see what they're up to. SmartSE (talk) 12:49, 2 September 2015 (UTC)
See Wikipedia:Sockpuppet_investigations/Tzufun for a contemporary example of what this could detect. It's ridiculous that it is so easy to evade detection by creating accounts that make 1 or 2 edits. SmartSE (talk) 13:00, 2 September 2015 (UTC)
  • (edit conflict)} This already happens - only 6 accounts can be created per IP per day (this can be overridden by administrators and account creators) - see Wikipedia:Account creator. I've been involved with several editathons where the limit has been hit, so the override is necessary for more than just the education programme. Thryduulf (talk) 13:02, 2 September 2015 (UTC)
  • Thanks for the link. As with anti-vandal measures, I guess we have to concede that there will be some false positives. As with our discussion at 11b it would only be necessary to log groups of accounts from IPs creating many accounts e.g. >10 in a month. It would be possible to distinguish good faith editors from socks very easily. SmartSE (talk) 13:16, 2 September 2015 (UTC)
  • An additional limit of X account creations per Y days could be very useful. It would need more discussion, e.g. does it need to be customisable (e.g. a university in September may see many legitimate creations)? Would spanning a range (to avoid IP hopping) cause more false positives than benefits? And it will almost certainly need a software update, but it's a good idea. Thryduulf (talk) 14:37, 2 September 2015 (UTC)
  • Support - I've also never understood why we allow 5 account creations at a time -- to what purpose? We have account creators if there's a legitimate need for a large number of accounts to be created. I'd limit account creation to 2, at most 3, at a time, and then a waiting period before any accounts could be created from that IP - say 7 days. BMK (talk) 21:00, 2 September 2015 (UTC)

11b. score articles on "sockiness"

Suggestion #11 is a good start, but should be part of a system that scores articles on "sockiness" or "COIfulness" based on a number of indicators. If the scores cross a threshold, the system would bring them to attention for human evaluation. You could even do it on a tool server hooked up to a Twitterbot a la CongressEdits and let the volunteer community self-organize around it. After spending some time at COIN, I have some ideas of what the scores should be based on, but this really deserves some serious thinking and testing of the scoring based on identified cases of undisclosed COI/undisclosed paid editing and socking. It should really be done as a machine learning system with partitioned learning/testing datasets, in essence datamining our own COIN and SPI datasets. As to elements of the scoring system, things I've noticed include:

  • Creation of fully formed articles on first edit
  • Creation of fully formed articles by new user
  • Creation of long articles by new user
  • Articles with certain types of content, e.g. business infobox, photo caption, logo
  • Approval of AfD by approver who has interacted with the submitter in certain ways. Categories of interaction to look at: interaction on any talkpage, interaction the submitter's or approver's talkpage, interaction on XfD pages, interaction on noticeboards.
  • Tweak interaction thresholds based on time between co-involvement on whatever page is analyzed.
  • Tweak "new user" thresholds

The above is just off the top of my head, if this gets traction I'll think of some more. Maybe a knowledge engineer could help us conduct good interviews of COI and COIN patrollers, and experienced admins. The neat thing about machine learning is you don't have to be perfect: you can throw a bunch of stuff into the training, and see what works. Poor correlators/classifiers will be trained out. For consideration — Brianhe (talk) 04:12, 2 September 2015 (UTC)

Other possible factors to weigh in a scoring system include:
  • External links quantity/quality metrics. Indicators include:
    • Multiple links to same domain
    • Social media links
    • Inline ELs in general
    • Inline ELs without a path
  • Distance to deleted article titles
  • Density of registered vs unregistered editors (sorry legit IP ed's)
  • Editor trust metrics (longevity, blocks, noticeboards, corpname patterns, ...)
  • There could also be a voluntary (opt-in) system for various actions that could modify the trust scoring for a particular editor:
    • having a WMF-verified identity on file
    • participation in a "ring of trust" system along the lines of WP:Personal acquaintances
    • community-bestowed reputation points
Another idea, if this scoring system worked sufficiently well, high scores could trigger automatic reversion of the article to the Pending changes protection model. — Brianhe (talk) 06:43, 2 September 2015 (UTC)
Thanks Brianhe, but I see that as a separate project targeted at auto assessing articles for spam much as we have bots and edit filters to deal with vandalism. This is a complex task for several reasons, but if we can come up with an algorithm that has few false positives it could be used for bot based spamfighting much as bots have brought vandalism under control. No-one nowadays would credibly predict that Wikipedia will eventually succumb to the rising tide of vandalism any more than they'd predict London would eventually overwhelmed in horse shit, but before the edit filters and vandalfighting bots that was as valid a prediction as the prediction in 1890 that London's traffic problems would bury the city in horseshit. Spam has been rising for years, I suspect it would eventually swamp the pedia if we don't find ways to automate dealing with it. ϢereSpielChequers 08:17, 2 September 2015 (UTC)
I'm exteremely supportive of this idea. As far as I understand, Cluebot works by giving edits a score based on how likely edits are to be vandalism and we could catch a great deal of promotional editing in a similar manner. We have plenty of SPIs listing confirmed COI accounts and we could presumably use these to train the bot to detect new edits. Special:AbuseFilter/354 Special:AbuseFilter/149 and Special:AbuseFilter/148 are the closest we have at the moment but they are comparatively crude and have barely changed in the last 5 years. I've not seen them catch any of the paid edits I've cleaned up over the last six months. How to get the ball rolling? SmartSE (talk) 12:46, 2 September 2015 (UTC)
You need to be careful here - what will be detected is promotional editing, not paid editing. Promotional editing is obviously harmful to the encyclopaedia, and improved detection of it is a good thing, but it will also catch unpaid promotional editing and will not catch paid editing that is not promotional. It is also incorrect to assume that all promotional editing by editors that have not disclosed they are being paid are breaking the terms of use - they may indeed not being paid and may just be a fan of whatever it is they are promoting. Thryduulf (talk) 12:56, 2 September 2015 (UTC)
That's the point though - content not contributor. If the content isn't promotional then there isn't so much of a problem (obviously this wouldn't deal with omissions etc. but it would catch a lot). Also, I should have said that unlike Cluebot I would prefer this to only tag edits for scrutiny rather than reverting them. SmartSE (talk) 13:13, 2 September 2015 (UTC)
@SmartSE:: Agree that none of this was intended to imply automated reversion, only tagging/reporting with human followup. At most, automatic reversion of the article to Pending changes protection based on certain tripwires. Actually I think we could start on this last one right away with fairly obvious parameters, like "new corp article by new editor". — Brianhe (talk) 18:34, 2 September 2015 (UTC)
You're pinging my doppelganger again ;) Does anyone have any clue who to ask about the feasibility of doing something like this? I noticed that neither of the original cluebot creators are really active anymore. SmartSE (talk) 20:50, 2 September 2015 (UTC)
Darn cut/paste. Brianhe (talk) 22:29, 2 September 2015 (UTC)

As an alternative to instituting a scoring system, consider using something like Amazon Web Services' Machine Learning to process the Wikimedia database, training it on known sock-posts, and letting it detect other candidates. Thanks to SmartSE for the heads-up, BTW. (Declaration of interests: I'm not paid by Amazon or AWS, nor did they ask me to comment on this, but I do - as an independent - run the Cambridge (UK) AWS User Group.) Jonsg (talk) 22:23, 2 September 2015 (UTC)

Have you done machine learning before? I have. You don't just throw a bunch of stuff at it and wait for an answer. You have to give it two things. One is the dataset itself. The other is scoring algorithms (including a way to read/parse the data). The individual criteria for the scoring, listed above, are implemented by a programmer as algorithms, which the machine learning then finds the best weights for in the overall score, that give you the best classification. Even for that, the definition of "best" is up to interpretation, because you have to decide what your acceptable false positive/false negative rate will be. — Brianhe (talk) 22:29, 2 September 2015 (UTC)

12. Write a Wikipedia article about paid editing

There's been plenty of reporting in the press, and this is definitely a notable topic. [9] A Wikipedia article that actually names abusive companies and explains their shady tactics could be a deterrent to the people who pay those companies. I like to imagine someone searching the internet either looking for somebody to write an article for them, or researching a company that has approached them, and finding our article near the top of their search results. ~Adjwilley (talk) 02:36, 2 September 2015 (UTC)

Conflict of interest editing on Wikipedia is pretty close. Ocaasi t | c 03:26, 2 September 2015 (UTC)
Indeed, it is. Even the major companies like Wiki-PR seem to have their own articles already. I do have another question: Are the checkusers able to link individual sockpuppets to the larger companies? That's actually two questions: do they have that ability, and would they be able to publicly disclose that, say, the orangemoody accounts are from such and such a company? ~Adjwilley (talk) 04:23, 2 September 2015 (UTC)
Regarding the Orangemoody socks, we do not know who is behind them. Speaking more generally, if there is sufficient evidence to be certain of a link between a sockfarm and/or paid editing ring and a corporation then I suspect that this will be publicly disclosed, although the advice of the WMF will be sought on each occasion. It is also possible that culprits will be identified through non-checkuser means, and I don't doubt there are people right now trying to do this based on the publicly available Orangemoody evidence. Thryduulf (talk) 10:34, 2 September 2015 (UTC)

13. Bonds for certain types of editing

Why not adopt a model from other areas and require the posting of a professional paid editor surety bond for good conduct? Bond would be forfeit fully or partially for various types of misconduct to be determined. This could be done in conjunction with one or more of the ideas above; for instance, I see it working well with #6 corporate accounts. A properly functioning market will price the bonds according to the editor's risk: a shady no-reputation actor would have to put up a lot of his own dough, but a reputable org or individual should be able to get a better rate. There would probably be some intrinsic tradeoff of privacy for accountability that the market would also see to. — Brianhe (talk) 06:54, 2 September 2015 (UTC)

We want to make legitimate interactions with use easy. And than we made to make poor paid promotional editing hard. Do not think this will accomplish either one of those. Doc James (talk · contribs · email) 07:39, 2 September 2015 (UTC)
Our aim is for a global service, to achieve that we need to tackle systematic bias not encourage it. A bond system on a global site would inevitably be set at a level that some corporate PR types would consider trivial, but in other parts of the world would seem extortionate. It would be less onerous on the full time wikipedia spammer than on someone who only spams about one or two companies. It would lead to court cases by people claiming they shouldn't lose their bond because what they did wasn't excessively promotional. But worst of all, it would squeeze out the retired, neutral and disinterested as they hadn't paid for the privilege of editing Wikipedia. ϢereSpielChequers 09:57, 2 September 2015 (UTC)
I should clarify, the bond was only intended to apply to "certain types of editing" by which I mean primarily paid editing, though we could extend it at will, e.g. all articles with MEDRS implications could require a bond. There should be zero cost to individuals who are not paid editors.
Doc, I think it would actually accomplish both. 1) Making poor paid promotional editing hard: There has to be a cost associated with bad content. Right now the cost is all born by the volunteers like us. The bond would be set to a monetary point, and with terms such that it should scare away poor promotional editing by those with a commercial interest. This is simply a way to make the creator of content have something on the line to offset the incentive that got them to write the content in the first place. Right now it's all a one-way system where the upside is writing content, and the downside is virtually nonexistent: damage to reputation of a throwaway account. 2) Making good-faith editing easy. This should not affect that side of the equation at all, if the good-faith editor acts appropriately. If they violate the terms they agreed to when they created their account and started editing, and re-affirmed when committing to the bonded editing process, then too bad.
@WereSpielChequers: I agree, the litigious US system could be against us in this regard. However this might be a consideration better made by WMF legal staff; a lot of the other proposals on the table here would require enforcement that would be at least as onerous on them. If this solution is what we-the-community want, we should ask for it.
Tweaks to this proposal could include:
  • Setting a very low bond value at first to demonstrate our resolve and to clarify process for good-faith people. Raise it over time to drive away the systematic exploiters.
  • Setting a variable bond level based on locality. I don't like this because I can see bad content being "offshored", but we may have to live with it for the perception of equity, if the alternative is no system at all.
  • Maybe membership in good standing in recognized professional organization (e.g. IEEE, AMA or even recognized PR associations) could be the equivalent of a bond for our purposes.
  • The system could be applied across the board, or selectively for certain types of articles. For instance, only to business-related articles or only to startup-related articles. Enforcement would be easy: delete on sight if the editor was in a prohibited editor group/topic area, and was not bonded.
Submitted for (re)consideration — Brianhe (talk) 18:13, 2 September 2015 (UTC)

14. Require real-name registration for editing and/or creating certain types of articles: small and medium-sized companies and organisations, biographies of lesser-known people

Looking at articles on small and medium-sized businesses (law firms, management consultancies etc.), the majority of them seem to have been created by a Wikipedia editor who has done little or nothing else in Wikipedia, suggesting it was a principal, employee or agent of the company in question. Same with other types of organisations and biographies of living people. These are typically articles that get relatively little traffic and scrutiny from regular editors – and the number of highly active editors per 1,000 articles is continuously shrinking (about a fifth of what it was in 2007). The problem will get worse, not better, over time. Andreas JN466 11:15, 2 September 2015 (UTC)

  • I strongly oppose requiring a real name on Wikipedia - even ignoring the swathes of privacy issues, we have no way of verifying what a person's real name is or is not. Read the Personal name article to see some of them problems with even defining what someone's "real name" is. Thryduulf (talk) 11:33, 2 September 2015 (UTC)
  • I understand why the PR industry wants this, but why should we concede it, especially just after catching some PR outfit out in a massive breach of the TOU? There are two forms of real name registration, one is that used on certain wikis which require your username to be plausible as a real name in the Anglosphere. The other requires verification against an external site such as Facebook. Both systems have the disadvantage that they are a greater deterrent to goodfaith editors than bad, the more effective the system at outing our goodfaith editors the more vulnerable it makes them to litigious companies who don't want anyone changing their PR department's desired wording of the Wikipedia article on them. Of course real name editing is a very different burden depending on how common your name is, and we might well have a few editors with names like John Smith who would still be able to edit PR vulnerable areas. But if we introduce real name editing largely the spammers would have won. ϢereSpielChequers 11:52, 2 September 2015 (UTC)
    • I am not sure the PR industry wants this. Certainly, the Orangemoodies are unlikely to want it. As for how to identify, see below (I had neither of the methods you mention in mind). Note too that this would only affect certain types of articles: in particular, those for small and medium-sized businesses, which few unconflicted people want to edit in the first place. Andreas JN466 17:30, 2 September 2015 (UTC)
      • If you require real name disclosure to edit such articles then you muck up the work flow of gnomes like me who eradicate particular problems across the wiki. Nowadays I have for various reasons verified my name with the foundation, but before I had would it have been reasonable to expect me to disclose in order to fix a typo in an article? I've recently been de arting various medical collages, frankly it would have been a pain if I had had to worry whether they were articles I was authorised to edit. As for whether the PR industry wants to identify our editors, about three years ago I was one of the volunteers who tried to explain the wiki side of things when someone in Wikimedia UK invited the PR people in to try and agree a code of practice. Absolutely they wanted to know who our editors were and they were quite put out when we explained that identifying and cultivating relevant editors wasn't the way to go. ϢereSpielChequers 20:27, 2 September 2015 (UTC)
  • I tend to agree with the above two opinions. Since we do require that paid editors disclose their employers, clients, and affiliations, you could say that we have real-name semi-registration for them. But if paid editors don't disclose the real names of their employers, etc. how could we force them to disclose (even in private) their own real names? There might be some use in brainstorming some ideas here - how can we force disclosure from paid editors? I'm not sure it would get far, but let m try 2.
  • Have the businesses and other payers register an account in their own names (see above proposal) and have them disclose the user name of who they are paying, together with submitting an OTRS ticket identifying themselves to prevent Joe jobs.
  • Require that paid editors enable the e-mail link, if serious issues have been raised, and confirm their identity to Arbcom or other people assigned this task.
Very likely those would be difficult to implement, e.g. just getting through an RfC would be difficult, but I think if we bounce around enough ideas we might come up with something. Smallbones(smalltalk) 16:29, 2 September 2015 (UTC)
One possible method often mentioned in this context is a credit card payment to the Foundation of £0.01. Identification to the Foundation is all that is required here: the account could still have a fantasy name on-wiki. However, the additional user right would only be given in response to the nominal penny payment. Andreas JN466 17:30, 2 September 2015 (UTC)
Problem: anonymous prepaid credit cards. — Brianhe (talk) 17:54, 2 September 2015 (UTC)
Another problem is that credit card payments cost the recipient to process, many places in the UK don't accept credit card payments for transactions below £10 because it's not economical for them to do so. $0.01 transactions would be a significant drain on the WMF's budget (international payments maybe more so) for at most trivial benefits. Thryduulf (talk) 18:02, 2 September 2015 (UTC)
Lots of people do not accept prepaid credit cards, so I suspect it's possible to filter them out. And if it costs £0.50 to process the payment (the amount many people in the UK charge for transactions below £10), the payment amount could be adjusted accordingly. Processing could be fully automated: People enter a user name that they want to purchase the user right for, and the system automatically adds the relevant flag. Andreas JN466 19:00, 2 September 2015 (UTC)
  • If this were implemented, everyone who creates stubs about, say, 7th-place Olympic gymnasts or Paralympic boccia medalists, or perhaps minor female scientists, would be required to out themselves or stop editing. I guarantee you that I would never have created what small number of articles I have if I had had to reveal my real name. BethNaught (talk) 17:58, 2 September 2015 (UTC)

15. Community participation in investigation tools

It's apparent from the Orangemoody case that the checkuser team is using some sophisticated case management tools (Maltego at least). Why aren't these available to trusted members of the community on an as-needed basis? This could dovetail nicely with the "evidence lockbox" idea I proposed under idea #5. For instance, I believe that many COIN volunteers have private stashes of notes, but there's currently no sanctioned way to share these. Obviously there'd need to be some protection to keep investigations private, and to create policy allowing this. — Brianhe (talk) 18:27, 2 September 2015 (UTC)

Yes please. I suggested something similar a few months back on User:Bilby's talk page but nothing came of it. I'd certainly like WMF to help us share suspicions more privately and share tips on how to track UPE. SmartSE (talk) 20:21, 2 September 2015 (UTC)

16. Ban all Admins

Excuse me, but I've always wanted to put in a section header like this in a very serious discussion.

My real proposal is "Ban all administrators, arbitrators, checkusers, and other functionaries from acting as paid editors."

Consider the following paragraph in the Guardian:

"The Wikipedia Foundation said the accounts were blocked over “black hat” editing – charging money for the creation of promotional articles – amid allegations that hundreds of businesses and minor celebrities have been blackmailed by scammers posing as Wikipedia administrators."

It would be very good if we could tell everybody - everybody at WP:AFC, at any meetings with businesses, at the Guardian or any other newspaper - everybody, that admins are *not* allowed to accept payment for any services on Wikipedia. Do not believe those who claim to be admins and ask for money.

This would be a bright line rule that would protect everybody involved, admins, the WMF, and the scam targets.

BTW, I am not accusing any admins of being paid editors. Smallbones(smalltalk) 18:43, 2 September 2015 (UTC)

  • Agree. Seems eminently sensible, for the reasons stated. People would have a choice: take money, or be an admin. Andreas JN466 18:50, 2 September 2015 (UTC)
  • Agree The potential for conflict of interest between a paid contract and administrative duties is not in line with an elevated position of trust in the community. Also, claiming to be an admin likely gives the paid editor a commercial advantage either in gaining clients or in what they can charge. This means that an admin gaining money from the community's trust and makes adminship a financial prize. Going forward this can be only be bad.

    A rule preventing admins from doing paid editing from their admin account or advertising as 'having access to an admin' etc is a great 'bright line' and by making it we can at the very least prevent such claims on Elance etc (Better to ban such freelance sites altogether but at lease prevent the monitization of the mop). I do not know if any admins are doing paid editing and trading on their status as admins but if they are they should stop. If they do not stop they should loose the mop. RfA is about who we trust to do the tricky stuff and the stuff requiring trust it is not about raising an editor's earning potential. JbhTalk 19:06, 2 September 2015 (UTC)

  • Agree as well. There was a point in my Wikipedia editing history (I almost wrote "career") where I thought about pursuing adminship, but decided that since I was beginning to represent clients, this would create the potential appearance of impropriety, and would be entirely avoidable if I didn't try for RfA. It was the right call. I'd be very supportive of seeing this made policy. WWB Too (Talk · COI) 19:33, 2 September 2015 (UTC)
Update, clarification: A distinction must be made between paid editing for outside interests and GLAM work with like-minded institutions. The language "paid editor" is not quite right here; "paid advocate" is better, because I doubt the intent of this proposal would be to prohibit admins from becoming WiRs. This would be a necessary carve-out. WWB Too (Talk · COI) 21:52, 2 September 2015 (UTC)
  • Agree. If we had to have a mechanism for temporarily suspending adminship—a leave of absence if you will—to accomodate and encourage compliance, I'd agree with that as well. Bottom line, active admins should not be paid editors and no admin should touch an article (or debate, etc.) that they have a conflict with, including one they previously worked on for pay. — Brianhe (talk) 19:43, 2 September 2015 (UTC)
    • Leave of absence might work for Wikipedians in Residence (unless they should be exempted). Andreas JN466 19:55, 2 September 2015 (UTC)
  • Agree How could anyone disagree with this? SmartSE (talk) 20:22, 2 September 2015 (UTC)
  • Disagree - The term "paid editor" is too blunt and imprecise to be meaningful in this context. It's COI that's a problem, not being "paid." -- Fuzheado | Talk 20:49, 2 September 2015 (UTC)
  • Agree - excellent idea, one that never had occured to me. BMK (talk) 20:51, 2 September 2015 (UTC)
  • Great idea Yes we have an account on Elance pretending to be this admin User_talk:Andrew_c#User:WayneAnde. It would help prevent impersonation. Doc James (talk · contribs · email) 21:44, 2 September 2015 (UTC)

17. Active promotion of Wiki ethics and integrity

I don't know what my concrete proposal is here, but I was shocked that #16 had to be said, I just assumed that ordinary ethics would have covered this already.

Perhaps a concrete part of this would be to have inclusion in an "ethical Wikipedians" society with strict guidelines and badge of membership. Kind of like the the statements and graphics below (see COIN convo on why this was necessary due to impersonation). If this was a thing, admins who didn't display it maybe would be shamed into either abandoning adminship, or signing up.

I do not edit or otherwise contribute to any WikiMedia article or project on behalf of any employer, client, or affiliated person, organization, or other entity; nor do I receive or solicit any compensation for any edits or other contributions.
This editor does not accept paid editing work. If somebody claims that he/she is me and is soliciting paid editing work, then they are impersonating me, and likely scamming you. Feel free to contact the proper authorities.
COI+This user follows the COI+ agreements. Talk to me if you think I need help with my editing!
This editor is a volunteer, and is willing to write and maintain encyclopedia articles for free.

Another concrete thing would be to revive WP:WikiProject Integrity and include reports on its activities in Signpost.

Another concrete thing would be to separate the Conflict of Interest Noticeboard active cases from general questions about what's right and what's wrong. Right now it's pretty punitive (I admit I'm part of that problem) and there's no place I know of to go with genuine questions of ethical integrity. — Brianhe (talk) 19:23, 2 September 2015 (UTC)

I agree with a lot of this. Not every detail, but the basic point is a good one (and similar to what I've just posted below) that spotlighting and and improving the experience of being a disclosed COI editor on Wikipedia will make it more clearly a viable path to follow. WWB Too (Talk · COI) 19:30, 2 September 2015 (UTC)

18. Make it easier for company reps to receive community help

Streamline efforts to assist responsible, hands-off paid contributors to improve community responsiveness to requests.

Speaking as one of the handful of disclosed, guideline-compliant paid editors around here, I'd like to offer the following: a common point of trouble for those who would do the right thing is that it can be very difficult to find a volunteer to assist with a request. I frequently hear from companies who have tried posting a request but found it confusing, or failed to receive a reply. The biggest issue isn't usually opposition to the requested changes—it's that they're hearing crickets.
An interesting comparison would be AfC. For all its problems, the fact remains it is an actively managed queue where one can eventually find the necessary assistance. Not so with Category:Requested edits, where there are unanswered requests from March. Sometimes one can find help at WP:COI/N although that is not its stated purpose. Once upon a time there was WikiProject Cooperation, but after some early support the volunteers dwindled.
So how about a concentration of efforts to make it easier for companies and organizations to take a hands-off approach. For lack of a better proposal, let's call this AfI: a content-focused help desk to which all company representatives can be directed, and expect a timely answer, whatever it may be. WWB Too (Talk · COI) 19:26, 2 September 2015 (UTC)
I'd be a lot more likely to help if I didn't have doubts about who exactly I was helping. I think as long as Wikipedia has the degree of anonymity that it does, I'm unlikely to devote my volunteer time to enriching another person. The only way this would work for me is if the editor asking for an assist was verifiably backed by a cause (e.g. a nonprofit) that I believe in. Maybe having individual WikiProjects adopt and endorse paid editors would go towards this? — Brianhe (talk) 19:35, 2 September 2015 (UTC)
I agree that anonymity is a major hurdle to building a trust relationship, which is why I use my real name on-wiki, and encourage clients to do the same. However, I see the exchange differently: consider that you're donating your volunteer time to help someone follow the rules as they offer improvements to Wikipedia about, yes, themselves. If the suggestions improve Wikipedia and benefit the requestor, so be it. If the suggestions do not improve Wikipedia, don't implement them, and explain why.
Meanwhile, having paid editors endorsed by WikiProjects is an interesting idea, but few projects are strong enough for this to make any sense. WP:COMPANIES itself is mostly tumbleweeds. WWB Too (Talk · COI) 19:47, 2 September 2015 (UTC)

19. Tighten up business-related content guidelines

Overhaul and tighten up the existing guidelines for business-related content.

Related to the point above about CorpProd (which, FWIW, I would be entirely in favor of a specific and strict notability standard for company articles), it would also be good to see a complete refresh and clarification of the guidelines for company articles. Anyone who works on a regular basis with company articles will know that the typical content of a company article, even those that have reached FA status, is hugely variable.
The WikiProject Companies guidelines are very basic and not terribly clear about what should be included, or excluded. Part of these very short guidelines also conflicts with the common sense feedback that often comes up at company articles about not wanting to include lists of executives; the guideline specifically says this is "desirable".
In short, having a clear guideline on appropriate content for company articles would give editors something to easily refer back to, as well as providing a resource to those seeking to update an article about their own company so that they understand what information can and should be included, as well as what should not. WWB Too (Talk · COI) 19:37, 2 September 2015 (UTC)

20. Increase the bar for autoconformation

As the initial report indicated, the socks created for this enterprise made their 10 trivial edits and waited the 4 days to be autoconfirmed. This is a ridiculously easy requirement. I would suggest at least 50 edits and 30 days, making it more onerous to create large numbers of autoconfirmed socks - and that standard is still easily reached by normal accounts. (Right now, IP exempt accounts editing through a Tor network are required to make 100 edits in 90 days.) BMK (talk) 21:06, 2 September 2015 (UTC)

See #3 above, as this is largely the same issue -- Fuzheado | Talk 21:30, 2 September 2015 (UTC)

Leave a Reply