feedback on test

I have taken several other tests since I posted last comment. The grammar and proofreading on all of them is dreadful. There are even obvious typos in the sample code!

The worst, though, was on the Windows NT test, where a question asked how long a "resourse" [sic!] would exist. There were several time values given as answers, but no units -- the question could have been referring to milliseconds or hours, or anything in between.

Having tests is a great idea, but someone really needs to edit them to a higher standard if they're going to be taken seriously. 

Hello HEI-USER,

Considering that every time someone takes a test, a new list of questions is put together (so that when re-taking the test, a test taker doesn't get to answer the same questions), what would be the most convenient way for you to assess the test and the provider's performance?

Would you like to be able to take  a (sample) test yourself?

Would you like to be able to see the test history (list of Qs and As the provider gave) for each provider/each test?

Would seeing 5 sample questions and possibly 5 answers the provider gave to these questions do the trick?

It is in our best interest to allow you to use oDesk test results to find a provider most suitable for your projects, but we do need more of your input to implement this successfully.

Olga Kudamanova

Provider Operations Manager

... what would be the
most convenient way for you to assess the test and the provider's
performance?

I'm not entirely sure as I haven't thought it through completely.  All I know is that having the raw score isn't enough to do anything for me.  I see some people with high nineties on a certain test, and others with 40-50%.  But, according to the other information, it looks like their skills would be comparable. 

That means that either a) one provider is lieing on their skill levels, or b) the tests aren't very fair or accurate.  From what I can tell, I'm suspecting b).

Would you like to be able to take  a (sample) test yourself?

That would probably be useful, in that it would give me an idea about the complexity and/or level of the test.  However, obviously since you randomize the questions, I can't be sure that all the questions on the test are fair.

By randomizing the test questions from a large question pool, you're losing the ability to compare provider to provider.  If I knew that every provider who takes the Java test gets asked the same questions, then I know I can compare the two candidates by their test scores.  However, if they each received different questions, then I can't compare apples to apples and there's no way to know that one candidate just didn't end up with all the harder test while another candidate ended up with the easier test.

Would you like to be able to see the test history (list of Qs and As the provider gave) for each provider/each test?

Yes, that would be ideal from a buyer's perspective.  However, it might cause some concern to your providers (potentially revealing too much information)?  I don't know, you'd have to ask the providers if they're comfortable with this.  OR, even better, provide an option on the test that the answers can be viewed by buyers, maybe after they've been selected for interview or something.  That way providers can decide if they want their answers revealed or not.

Would seeing 5 sample questions and possibly 5 answers the provider gave to these questions do the trick?

Possibly, though I don't know how this would be different than seeing all of the answers.

Probably even better, it would be nice to see the questions which the provider got wrong.  That way, I can see if the answer seemed tricky and if the test taker possibly provided a reasonable answer.

Again, maybe this information is available only after the candidate has been selected for an interview.  Or maybe it's only available if the provider allows it to be.

It is in our best interest to allow you to use oDesk test results to
find a provider most suitable for your projects, but we do need more of your input to implement this successfully.

I know this is completely a different idea, but in my mind it would be really nice to be able to create a test for the provider to take.  I could look through the pool of all questions and create a specific test per job opening.  That way, I can test across all the technologies that I require for the job.

For instance, with the job I'm currently fielding, it would be ideal to have some sort of test which measures: Java basic, Java Servlets, Hibernate, SQL, J2EE, HTTP, JFC/Swing.  If I could put together a test which I could ask questions across all of these categories, that would be pie-in-the-sky ideal for me. But, I know that's a different idea entirely.

Also, while I've got your attention, just reading these forums, it's evident to me that the test taking procedure is flawed.  Specifically, the inability to go back and correct answers is a huge testing methodology mistake.  A good test will always allow the taker to answer any question in the order which they want.  The test will yield more accurate results this way.  The inability to go back and change answers before the time has expired is a fatal testing methodology.  It's likely one of the biggest reasons why test scores are relatively low across all test takers.

One final thought.  I'm not sure how it works, but I know if I were a provider and I took a test and got a low score on it, I would probably rather not have that test score revealed as it would likely be more detrimental.  I've seen low scores on a lot of tests taken by providers, and it's hard as a buyer to not be influenced by that test score.  Again, because I can't get any feedback on why the test score is so low, I can't tell if the candidate really did perform badly, if the test was written poorly, or what.  If I tested low, I know I would either want a chance to retake the test or to not have the score revealed to the buyers.  If the providers don't get this chance, it might very well ruin their ability to get any work at all.  Eventually they'll log off and never come back.

I think one of the best ideas discussed above is to give more options to the provider to allow them to choose whether or not their scores are revealed, and whether or not their individual questions/answers are revealed.  That would give the most protection to the providers and the most transparency to the buyers.

Thanks for reading.

Thank you very much for this detailed answer!

First of all, let me confirm that currently, the Providers can both make their test scrore private (that is choose to not show it in their Profile) and re-take a test after a period of time (normally after 30 days). 

The test score is visible in the profile by default, so a provider needs to take an action to hide them, but the possibility is right there for them.

The 30 days waiting period was introduced to make sure the Provider doesn't abuse the  system and studies some before re-taking the test.

As far as your suggestions go, let us digest them for while and get back with what can be implemented and the best way of doing so. Again, thank you so much for taking the time to think them over and post here. It is immensely helpful for us.

Regards,

Olga  

Hei-user,

To add to Olga's comments:

 1. The reason we have disabled a test-taker to go back and edit  previous answers is to prevent cheating. We dont want providers to be able to "'park" the questions they dont know the answer to, and then google around for the answers and come back and answer them. This helps to preserve the integrity of the test scores.

 2. The test randomization also helps in preserving the integrity of the tests. Since no two instances of the test are the same, a single provider cant improve his/her test score by taking it repeatedly, nor can he/she help his friends by distributing the questions to them. The testing engine makes sure that the "difficulty level" of 2 instances of the test are relatively the same. So you should be able to make an apple-to-apple comparison between 2 providers who have taken a test.

Please take a look at the FAQ and the policies for online testing program here: online_testing_overview *link updated by Jacqui P

4. It's important to note that the tests and the scores should NOT be the only criteria you use to make hiring decisions as a buyer. Yes, it should definately be one of criteria to evaluate whether the provider really posesses the skills he/she mentiones on the profile. But other criteria such as the provider's feedback rating, the cover letter, your interview with him/her...those criteria should also play into your decision making.

5. The ability for buyer to create a "cocktail" test of his/her own choice is something that is definately a possibility we can consider in the future. Also, the ability for a buyer to take tests, right now only providers can take tests.

If you would like to take the tests yourself in the mean time, please sign up as provider here:

http://www.odesk.com/professionals.php

You can take any of the tests after you get  your provider account.

Thanks,

Sawan

oDesk Corp.